Mind Dump

Rail 3.1 CI setup with Jenkins, Test Unit & SimpleCov on OS X Lion.

2011-09-25T11:54:00.000-05:00

I recently had to setup a build server for some rails work I'm doing. Still wanting to support my other projects I setup Jenkins. I ran into several issues.

Running Jenkins as a hidden user

First I noticed that jenkins was running as the "daemon" user, this obviously wasn't going to work for github and rvm needs. So I did some googling and had some guides to get Jenkins running as a specific user. I did the following (sourced from http://colonelpanic.net/2011/06/jenkins-on-mac-os-x-git-w-ssh-public-key/ ).Note: That's really $PASSWORD up above. This gives you a prompt to enter that password. Next you'll need to stop the Jenkins service and edit the plist and start the service back up.
You're plist file should end up like this.

RVM issues

Now my next issue was despite what I'd read elsewhere I was unable to get Jenkins to use the default ruby provided by RVM. So I just pasted the commands that I would run anyway in the "Execute Shell" build step.

Getting Jenkins to see tests

I've been using Test:Unit/Minitest lately just to keep more consistent with my day to day work. However I haven't found a way to get my tests to show when using the "Execute Shell" task. I found a little gem called ci_reporter that exports to the standard junit format, unfortunately it doesn't work with minitest yet. That's ok I haven't done anything that Test:Unit doesn't support so far so I added the the following to my Gemfile (note the part about unit-test 2.0):
Running "rake ci:setup:testunit test" should give you a bunch of xml files in tests/reports. Now we need to tell Jenkins where to find those reports so add a post build action to pick them up as junit reports.

Rcov reports

This was pretty easy.

Install Jenkins plugin for RCov (it's in the plugin list in the admin section).
add simplecov and semiplecov-rcov to your Gemfile.
configure Jenkins rcov plugin to look in coverage/rcov
add the following 4 lines to the TOP of your tests/test_helper.rb file:

In closing

This took a fair amount of time, but the end result was quite satisfying. I now have CI with tests, tests coverage, RVM to target different versions of Ruby and bundler to make sure my Gem environment is sane.

The difficult definition of professional software development

2011-02-04T16:49:00.000-06:00

Here are some of the contradictory phrases (and a few paraphrases) I've overheard used to define what is "good" and "bad" code.

Code should always be well commented
Maintainable code has unit tests and well named methods therefore needs little if any comments
Class explosion is to be avoided at all costs
Many small simple well named classes are the key to well organized code
Unit tests really don't test anything useful
Dependency injection leads to an unusable mess
Dependency Injection is a very useful tool for making your code extensible and easily reused"
Inheritance leads to solid code reuse
Inheritance is the strongest coupling of your code you can have
Functional programming simplifies and minimizes the complexity of your code
Functional programing just makes my eyes bleed!
Static global references are 'just programming' and something you have to learn how to manage to make simple easy to understand code
Static global spiderwebs are crippling to maintenance and program lifetime

The part that is most difficult about that list for me is while I agree with half of it strongly and respect the people who said those things, the other half that list I do not agree with, has all come from people I respect and who overall do some pretty impressive things. They've certainly created things by all measures more impressive than anything I've ever produced

This leads me to question how much do software principles matter when taking them out of the context of yourself but viewing it in a bigger picture? Thinking about it, 95% of the software that I actually like probably wasn't using any TDD at any point in time and certainly violates a number of things that I would consider required for "professional software". I keep reinforcing this fact every time I checkout the source of a major software project I've used for years and gasp in horror at the spaghetti code I find.

Worse still the different schools of thought are not compatible in the slightest, and one side views the other side as wholly unprofessional (granted for different reasons) to the point that I've realized I myself was perceived as the one being "amateur" by those with a different definition of what makes good and bad software. I of course unfortunately often thought the same of them, regardless of how I felt about them personally.

Anyway, this is all food for thought and I have not yet come to any conclusion what it all means. I know I've tried coding under other schools of thought and while through practice I was able to deliver well enough, I'm far slower and more error prone with no TDD, big mega classes, and avoiding dependency injection.

Java IoC containers and classpath scanning (or what I’ve been looking for from .NET for months)

2010-09-19T14:01:00.001-05:00

Frustrated with the typical way I saw IoC used in Java where every example I found involved thousands of lines of XML
and/or Java code to configure Java beans or components. This is very different than IoC typically used in .NET where most IoC containers allow
you to "autowire" in their terminology up every class in an assembly with a couple of lines of code. Having been coding in that fashion
for several years in .NET I was dismayed when none of my fellow Java coders that I worked with or knew personally had any concept
of the equivalent functionality, and instead informed me the IDE would be my help in maintaining these massive XML files.

Not content with their answers I burrowed into Spring, Guice and PicoContainer docs and found "classpath scanning" which is roughly
equivalent to "autowire" in .NET. Below is an example of this in Spring 3:

MyStuff.java (my main class note his has 2 dependencies which you can view on my github repo)

package org.foo;

import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.stereotype.Service;

@Service //one way of marking this as a component to register

public class MyStuff {

    private final DependencyWithNoInterface first;

    private final DependencyInterface second;

    @Autowired //tells spring which constructor to use for it's dependencies

    public MyStuff(DependencyWithNoInterface first, DependencyInterface second) {

        this.first = first;

        this.second = second;

    }

    public void run(){

        System.out.println("foo me");

        first.foo();

        second.superFoo();

    }

}

appContext.xml (seems you still need some XML)

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans"

       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

       xmlns:context="http://www.springframework.org/schema/context"

       xsi:schemaLocation="http://www.springframework.org/schema/beans

           http://www.springframework.org/schema/beans/spring-beans-3.0.xsd

           http://www.springframework.org/schema/context

           http://www.springframework.org/schema/context/spring-context-3.0.xsd">

     <context:component-scan base-package="org.foo"/>

</beans>

Application.java (the initialization)

package org.foo;

import org.springframework.context.ApplicationContext;

import org.springframework.context.support.ClassPathXmlApplicationContext;

public class Application {

    public static void main(String args[]){

        ApplicationContext ctx =   new ClassPathXmlApplicationContext("appContext.xml"); //the IoC container

        MyStuff stff = ctx.getBean(MyStuff.class); //my fully injected class

        stff.run();

    }

}

Now I’m sure for the .NET developers familiar with StructureMap, Windsor, Autofac, etc this is completely unimpressive. I’m also sure there are Java developers that are somewhat unimpressed with this but for reasons that involve holding onto giant configuration artifacts. Manual IoC component registration falls where hand writing SQL for trivial data access does for me, extra repetitive work that has been solved years ago. I worked in factories a decade ago, and therefore now have a low tolerance for similar activity as a professional.

Edit:

It appears you can forgo XML as well in your main class use a different Application context and add the refresh and scan lines

package org.foo;

import org.springframework.context.annotation.AnnotationConfigApplicationContext;

public class Application {

    public static void main(String args[]){

        AnnotationConfigApplicationContext ctx =   new AnnotationConfigApplicationContext(); 

        ctx.scan("org.foo"); //scans the org.foo package

        ctx.refresh(); //needed to load them for some reason

        MyStuff stff = ctx.getBean(MyStuff.class);

        stff.run();

    }

}

Anti-Pattern: Too much of your application is about interacting with external resources

2010-07-15T07:00:00.001-05:00

Firstly, what am I talking about? Applications that meet some of the following descriptions:

Stored procedures with a fair amount of conditional logic or complicated business rules buried in a sub-query (some would argue sprocs at all).
Web pages with lots of conditional output again encoding business rules in snippets of view logic. if else checks that display the latest price for something if before and certain date or a default price.
The majority of ‘unit tests’ require a database, or web and application server to be up and running.
The majority of code files make reference to language default I/O or database libraries.
Hours are spent trying to determine if differences in version of infrastructure are the cause of certain bugs.
Changing database schema results in hours of refactoring, as hundreds of querys and sprocs are hunted through to see if this index or that index is being properly hit.

Why is this an anti-pattern? First before I bore you with endless reasoning behind SOLID principles, proper OO etc if you’re reading this and you’re of a bent that disagrees with me so far entirely..none of that will mean anything to you, and none of it will mean anything to your customers so let me put this in the most practical terms I can.
It is not testable in the slightest in any practical sense. You will be counter with well that with that sort of application and making everything a front to back test that you KNOW when things are working, yes but you RARELY if EVER know in a quick sense why things are not working. If i can reliably eliminate any chance that my actual code or business logic is the cause of a problem, if all of my fancy business rules are in something that can be run thousands of times a second, then I can throw all sorts of corner cases at my code base and not increase my verification time. If the majority of my application is business logic and plain old code then I can save my slow manual testing to the areas that are so bullet proof and only have a slice of my application doing front to back testing to yes, make sure things really work.
Note: Cross posted from Polyglots R Us.

Permalink

Project Management in Java: A Confused .NET Developer’s Perspective

2010-07-10T12:50:00.003-05:00

When I was first introduced to workplace Java the amount of ways one could define a project appeared to be restrictive, confusing and a point of frequent friction. While those things may all be true, it’s a great deal better than I would have expected since I was coming from a world where an upgrade to a newer version of Visual Studio requires your whole team to sync up to that same version! So how do cross IDEA teams get any work done in the Java workplace? Do Java OSS projects require everyone uses Eclipse, NetBeans, or IntelliJ? Does every new IDE need to have import/export filters for all of the other existing ones and if it becomes popular do the already present IDE’s have to include this? Which system is the best and what matters?
To answer these questions let break down what makes a project and while doing so compare and contrast the Java perspective with the .NET one.

Compilation

Java IDE’s  (Intellij/Eclipse/Netbeans) – basically run “make” on the fly and pass everything to javac.
JAVA IDE-agnostic project systems  (Maven, Gradle) – adding all the references contained directly to the classpath and then calling javac.
.NET (Visual Studio,MSBuild)  – MSBuild feeds all references to csc.
In summary, there is more similar than different here.

Strong Versioning

Java IDE’s – at least in the case of IntelliJ I ended up manually writing a MANIFEST.MF file.
Java IDE-agnostic project systems – still manual editing a lot of the time but usually just one line already filled out by your template that you edit. I know NetBeans gives you a GUI to edit this.
.NET – GUI but you can manually edit an assembly.cs file.
In summary, Java IDE’s at least in the case of IntelliJ really are using default Java by hand stuff.

Dependencies

Java IDE’s – All use convention! Basically, search for jar files in your source tree, if the IDE finds them it assumes you need them and adds them as references in your project format. Works pretty nice on a whole and allows for manual tweaking in the oddball cases.
Java IDE-agnostic project systems – requires explicit dependency reference by version, however they will download them automatically from remote locations and put them in your file-system in a specific way (defined by them unless overridden by you). Since both of these are also responsible for your compilation you have to basically run everything through them or your IDE has to be aware of them.
.NET – requires explicit reference to DLL’s on the file system.
In summary, .NET and the Java IDE-agnostic systems are VERY similar here in approach, whereas the Java IDE’s take a convention over configuration approach. I’d say there are times I definitely prefer the Java IDE approach and there are times the capabilities of the Java IDE-agnostic systems are worth it.

Source References

All Java solutions – do reference by directory structure and wild card matching.
.NET – explicit references to each class file .
In summary, this is the HUGE difference. Java takes more of a Make based approach where it’s EXPECTED this will not be the only IDE/project system ever used on this source tree, and this is where the Visual Studio/.NET experience starts to fall apart when trying to use cross-ide’s or versions. It has forced everyone to implement and keep up with MSBuild’s format to enable other IDE’s to be used, and don’t talk about actual alternatives to MSBuild. At this point even though things like NAnt and Rake/Albacore are more popular among agile .NET teams than scripting MSBuild almost all of those scripts still call MSBuild to do the compile step!

Questions Answered

Q. How do Java OSS projects deal with all the different IDE’s out there and even different project formats?/How do cross IDE teams get work done?
A. They don’t have to worry about it as much as .NET devs. A friend of mine who a lead contributor to the Apache Cassandra project in fact included a Maven file for the “enterprise types” but by and large use Ant’s to do their builds and he himself has used a number of IDE’s on that same codebase. .NET OSS development is still a bit hamstrung by MSBuild and really alternative IDE’s are keeping in sync with that project management format. There are real problems with style and code format guidelines, but there are usually workarounds or ways of solving this if code style guidelines is something you really care about ( which I don’t).
Q. Which system is the best?
A. There is no one size fits all answer here. I now know Apache Maven solves some problems for people and even though I don’t appreciate it’s approach to things, it’s done some valuable good for the community. The IDE-centric approach really works pretty well a lot of the time for projects. Gradle, while being the closest to what I want is very young and has limited IDE support, at least it’s not easy enough for this Java n00b to figure out how to use with IntelliJ without some friction.

Summary

I hope that all wasn’t too confusing, it sure was for me to go through while I had all these options , and in a completely different environment to what I was used to. Worse a lot of the folks I talked to saw these IDE-agnostic project systems as solving problems they didn’t have. Maven, Ivy, etc was a world that was buzzword laden (I’m really looking at you Maven and Ivy), involved heavy setup time for relatively simple tasks, incomplete documentation (looking at you Gradle) and was frequently badly misused (those who know me know what I’m referring too here). In the end an IDE-centric approach in the Java world does not seem to tie you to that IDE in any way I can find, and for getting started had little friction, and for people new to Java it seemed to be the best way to get started. Besides in Java at least it’s easy to change your mind later.
Note: Cross posted from Polyglots R Us.

Permalink

MySQL 5 Performance Tuning Toolkit

2010-07-01T07:00:00.000-05:00

Recently we’d played with table partitioning and because of the limitations of it and some decisions we’d made a very long time ago we ended up spending a couple of days tracking down hotspots. In the process I picked up a few tricks, I’m not even an authority on any of this but I figured it’d help out others in the same spot.

Logging Toolkit

If you have a slow loading page or test and have no idea which one of the thirty or so queries going on is the cause how to do you find it. Slow query log and logging queries that are not using indexes. Adding the following to your my.ini, my.cnf or whatever your mysql config file is named on your system and you’ll be able to get a list of offenders to further investigate.

MySQL Console Toolkit

SHOW INNODB STATUS – gives tons of useful information on what the current state of deadlocks, transactions and all sorts of information.
SHOW FULL PROCESSLIST – shows the current threads and queries running on them. running this over and over again can show you queries that are hanging.
EXPLAIN <QUERY>– Just in place of query type your full query statement, particularly in sub-queries, this is great for pointing out potential problems, in my most recent case I had an uncacheable sub-query, pointing out quickly something that was just not going to work no matter what.

Final Notes

Anyway this is only a smidgen of the useful stuff you need in your toolkit and I’m certainly no expert. I want to plug High Performance MySQL: Optimization, Backups, Replication, and More and the http://www.mysqlperformanceblog.com/ for it’s help and as a reference for those wanting to go into deeper study.
Note: Cross posted from Polyglots R Us.

Permalink

Hibernate Connection Pooling: why isn't the default one for production?

2010-06-26T08:53:00.001-05:00

Hibernate unlike NHibernate comes with a variety of connection pooling options. The three primary ones of which I'm aware are Proxool, Apache DBCP, and c3p0 . I myself have only so far used c3p0 and it works quite well having saved me from a couple of jams so far where the default one was in use previously. First why not use the default one? Because the official Hibernate documentation says in a big yellow box marked caution the following words:

The built-in Hibernate connection pool is in no way intended for production use. It lacks several features found on any decent connection pool.

Yet on almost every single Java project I've worked on the default is in use. I wouldn't care so much but this has personally wasted a fair amount of time for me and I've had to approach this as an outsider every time getting into the argument "it hasn't cause any issues before". Yet usually within a couple of days of that discussion I or someone else has resolved a "well it just does that sometimes" problem just by switching the connection pool. However, in implementing an alternate connection pool I should share a couple of gotcha's

1) Turn on logging for Hibernate and read the logs and make sure your connection pool of choice is actually activated and not the default one.

2) Older examples of the documentation I found for c3p0 did not include the necessary line for the C3P0Connection provider. If this line is not included in version 3.x version of hibernate regardless of what else is configured hibernate will use the default connection pool.

3) If the class you specified for connection provider line is not found for some reason, also the default connection pool will be used. Depending on your version of hibernate you may have to reference a different jar.

Below is the config borrowed from the hibernate connection tutorial only with c3p0 configured, please borrow it instead of using the default one.

<div class="line" id="LC1"><span class="cp"><?xml version='1.0' encoding='utf-8'?></span></div><div class="line" id="LC2"><span class="cp"><!DOCTYPE hibernate-configuration PUBLIC</span></div><div class="line" id="LC3"><span class="cp">        "-//Hibernate/Hibernate Configuration DTD 3.0//EN"</span></div><div class="line" id="LC4"><span class="cp">        "http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd"></span></div><div class="line" id="LC5"><br></div><div class="line" id="LC6"><span class="nt"><hibernate-configuration></span></div><div class="line" id="LC7"><br></div><div class="line" id="LC8">    <span class="nt"><session-factory></span></div><div class="line" id="LC9"><br></div><div class="line" id="LC10">        <span class="c"><!-- Database connection settings --></span></div><div class="line" id="LC11">        <span class="nt"><property</span> <span class="na">name=</span><span class="s">"connection.driver_class"</span><span class="nt">></span>org.hsqldb.jdbcDriver<span class="nt"></property></span></div><div class="line" id="LC12">        <span class="nt"><property</span> <span class="na">name=</span><span class="s">"connection.url"</span><span class="nt">></span>jdbc:hsqldb:hsql://localhost<span class="nt"></property></span></div><div class="line" id="LC13">        <span class="nt"><property</span> <span class="na">name=</span><span class="s">"connection.username"</span><span class="nt">></span>sa<span class="nt"></property></span></div><div class="line" id="LC14">        <span class="nt"><property</span> <span class="na">name=</span><span class="s">"connection.password"</span><span class="nt">></property></span></div><div class="line" id="LC15">        <span class="nt"><property</span> <span class="na">name=</span><span class="s">"connection.provider_class"</span><span class="nt">></span>org.hibernate.connection.C3P0ConnectionProvider<span class="nt"></property></span> <span class="c"><!-- NECESSARY TO USE C3P0 --></span></div><div class="line" id="LC16">        <span class="nt"><property</span> <span class="na">name=</span><span class="s">"c3p0.acquire_increment"</span><span class="nt">></span>1<span class="nt"></property></span> </div><div class="line" id="LC17">        <span class="nt"><property</span> <span class="na">name=</span><span class="s">"c3p0.idle_test_period"</span><span class="nt">></span>100<span class="nt"></property></span> <span class="c"><!-- seconds --></span> </div><div class="line" id="LC18">        <span class="nt"><property</span> <span class="na">name=</span><span class="s">"c3p0.max_size"</span><span class="nt">></span>100<span class="nt"></property></span> </div><div class="line" id="LC19">        <span class="nt"><property</span> <span class="na">name=</span><span class="s">"c3p0.max_statements"</span><span class="nt">></span>0<span class="nt"></property></span> </div><div class="line" id="LC20">        <span class="nt"><property</span> <span class="na">name=</span><span class="s">"c3p0.min_size"</span><span class="nt">></span>10<span class="nt"></property></span> </div><div class="line" id="LC21">        <span class="nt"><property</span> <span class="na">name=</span><span class="s">"c3p0.timeout"</span><span class="nt">></span>100<span class="nt"></property></span> <span class="c"><!-- seconds --></span> </div><div class="line" id="LC22">        <span class="c"><!-- SQL dialect --></span></div><div class="line" id="LC23">        <span class="nt"><property</span> <span class="na">name=</span><span class="s">"dialect"</span><span class="nt">></span>org.hibernate.dialect.HSQLDialect<span class="nt"></property></span></div><div class="line" id="LC24"><br></div><div class="line" id="LC25">        <span class="c"><!-- Enable Hibernate's automatic session context management --></span></div><div class="line" id="LC26">        <span class="nt"><property</span> <span class="na">name=</span><span class="s">"current_session_context_class"</span><span class="nt">></span>thread<span class="nt"></property></span></div><div class="line" id="LC27"><br></div><div class="line" id="LC28">        <span class="c"><!-- Disable the second-level cache  --></span></div><div class="line" id="LC29">        <span class="nt"><property</span> <span class="na">name=</span><span class="s">"cache.provider_class"</span><span class="nt">></span>org.hibernate.cache.NoCacheProvider<span class="nt"></property></span></div><div class="line" id="LC30"><br></div><div class="line" id="LC31">        <span class="c"><!-- Echo all executed SQL to stdout --></span></div><div class="line" id="LC32">        <span class="nt"><property</span> <span class="na">name=</span><span class="s">"show_sql"</span><span class="nt">></span>true<span class="nt"></property></span></div><div class="line" id="LC33"><br></div><div class="line" id="LC34">        <span class="c"><!-- Drop and re-create the database schema on startup --></span></div><div class="line" id="LC35">        <span class="nt"><property</span> <span class="na">name=</span><span class="s">"hbm2ddl.auto"</span><span class="nt">></span>update<span class="nt"></property></span></div><div class="line" id="LC36"><br></div><div class="line" id="LC37">        <span class="nt"><mapping</span> <span class="na">resource=</span><span class="s">"org/hibernate/tutorial/domain/Event.hbm.xml"</span><span class="nt">/></span></div><div class="line" id="LC38"><br></div><div class="line" id="LC39">    <span class="nt"></session-factory></span></div><div class="line" id="LC40"><br></div><div class="line" id="LC41"><span class="nt"></hibernate-configuration></span></div>

view raw c3p0.hibernate.cfg.xml This Gist brought to you by GitHub.

Dynamic DNS with Amazon EC2 Linux and EveryDNS

2010-06-06T13:05:00.001-05:00

So I finally sat down and did the math and found out Amazon EC2 was quite a bit cheaper than what I’d been paying for hosting as long as I was willing to prepay for at least a year. However, with EC2 you are getting a dynamic ip’s so what to do? Well I’ve been using EveryDNS for years for my dns hosting and despite it’s recent acquisition by Dyn, Inc it’s still works the same. The scripts that are described below can be downloaded here, what follows is a brief walkthrough.

Download EveryDns update script from http://www.everydns.com/dynamic.php and place it in something like /usr/local/bin.
Create a convenience script in /usr/local/bin that will get your external ip and have it contain the following lines and call it external-ip.sh . This is using Amazon’s suggested curl command to find out your EC2 instance Ip address.
Then create a script for the actual dynamic DNS update in /usr/local/bin/ and call it updatedns-sh. This is using the set command to grab the output of external ip and storing it in the variable $1, and then passing that value to the update command. NOTE: this will update the dns record of ALL your domains with everydns set to dynamic. If you need to use specific domain names there is a –d flag for the eDNS client.
create a cron job on your linux distro of choice but the cron job line should look like the following.

Projects in Java with Maven 2

2010-05-22T21:56:00.001-05:00

For those of you who don’t know Maven is a build tool/dependency manager/project model. Those in the Microsoft space can probably imagine MSBuild + the ability to download all dll’s for you.

What I liked

Dependency Resolution. Automatic downloading of dependencies rocks in concept. Just specify library name and version and when you compile again it’s there. No need to check in jar’s into your source tree.
IDE Independence. In practical terms lets you use whatever IDE you want with no import/export. Intellij, Eclipse and Netbeans all understand maven as a full project format. So your team can all have different IDE’s and not create havoc with one another.
Good Default Project Structure. Tests are in the same location by default, resources for tests and your prod code are in expected places.

What I hated

Inefficient with already downloaded dependencies. It will check remote repositories every time you build even when you specified version number. Now I can see why they did that but not as a default. I mean sure version 1.12 may have had a bad bug and the project hotfixed in a new one with the same version number, but I’d say that’s really unlikely. In a larger project with several repositories the difference in time to build between this and an alternate dependency manager Apache Ivy is stark.
Very opinionated view of the build process. Those of you used to (N)Ant or Rake will miss the lack of control. The AntRun plugin will help mitigate some of the lose of control.
HOME/.m2/settings.xml . Specific system wide settings do not belong in a build language. The bad heavy friction ideas that it enabled were legion. Just one example was I had to connect to my work VPN even when working on my home OSS projects hosted on github.
Way too XML for for sometimes simple things. To specify the version of java target language version see the figure below, this specifies Java 1.6.

<build>

  <plugins>

    <plugin>

      <artifactId>maven-compiler-plugin</artifactId>

      <configuration>

        <source>1.6</source>

        <target>1.6</target>

      </configuration>

    </plugin>

  </plugins>

</build>

Summary

If you have a cross IDE team or are running an OSS java project I find it very hard to ignore Maven. If you are looking for easier dependency management, less verbose build files, and do not care about IDE independence I suggest any number of alternatives such as Apache Buildr, Apache Ivy, Gradle or many others I’m sure I’m forgetting.

Java Dependency Management with Apache Ivy

2009-06-29T09:00:00.000-05:00

Not wanting to ditch your already built well working ant scripts for the plugin-centric view of Maven, especially if your project structure doesn't line up quite right with Maven's point of view? Enter Apache Ivy which like Maven can automatically download and resolve all of your dependencies and their dependencies for you with just a few simple lines of XML (if only I could get rid of the XML Part).

Pros

Integrates with Ant, but can still be run standalone.

By default dumps all dependencies into your lib folder which makes it super easy to use with most IDE's.

Works with existing Maven 2 repositories, allowing you to leverage work already done by the Maven community.

Cons

Despite lots of documentation, too much focus on theory with little practical example of how to do simple things. Maybe I'm missing something but I did waste a fair amount of time trying to get a custom 3rd party repository working without crippling the defaults for Ivy. Again I may be just dense, but this was harder for me than either Gradle or Maven to do, and I ran into a couple of hiccups from there.

Limited IDE support compared to say Maven at least in the case of Intellij. Part of this is Maven has some pretty snazzy integration with Maven and I know on more than a couple of occasions I've just started projects with Maven because Intelij makes it super simple and gives me lots of nice integration for adding dependencies with the classic alt+enter .

I decided to include the default scripts I start with. This should get you up and running and give you a template for solving some of the hassle I went through. Just place next to each other in your top level project directory, ivy.xml, ivysettings.xml, and build.xml if you plan on using Ant integration. Also for ant integration make sure you have ivy-2.1.0.jar in "$ANT_HOME/lib" and all should be well. Please ask for clarification on anything, I'll do another post if I get enough questions.

Over at los techies now

2009-05-22T11:05:00.003-05:00

I got a wonderful opportunity to blog over at Los Techies . I'm not too worried about losing my independence as I see eye to eye with a lot of that community and its a good place to get my idea's kicked around and adjusted by others experiences.

Context/Spec style testing and my approach to BDD

2009-05-21T09:24:00.003-05:00

I borrow heavily my approach to testing from a combination of Ayende's Rhino Tools tests, and my reading of the Rspec beta book. But I think I've stumbled onto something I'm happy with and I can generate reports out of. Let's go over some basic rules first:

Move as much common setup logic to a base class as possible.
Use your class name as the context
methods are rules beginning with "should"
create a new subclass of the base context every time you have a new scenario

Code ends up looking like so:


public class BaseAddVacationContext
{
    protected AddVacationRequest submission;
    protected IEmailSender _sender;
    protected IUserInformation _information;
    protected ICrudRepo<LeaveRequest> _leaverepo;
    protected LeaveRequest request;
    [SetUp]
    public virtual void SetUp()
    {
        _sender = MockRepository.GenerateMock<IEmailSender>();
        _information = MockRepository.GenerateMock<IUserInformation>();
        _leaverepo= MockRepository.GenerateMock<ICrudRepo<LeaveRequest>>();
        submission = new AddVacationRequest(_sender, _information, _emprepo);
        request = new LeaveRequest() { UserName = "james" };
  
   
    }
}
[TestFixture]
public class SpecAddVacationRequestWhenHappyPathOccurs : BaseAddVacationContext
{


    [SetUp]
    public override void SetUp()
    {
        base.SetUp();
        _information.Stub(x => x.GetManagersEmailAddresses("james")).Return(new[]{"jacob@jonbank.com", "johnny@jonbank.com"});
        _information.Stub(x => x.GetUserEmail("james")).Return("james@jonbank.com");
        submission.Execute(request);
   
    }
    [Test]
    public void should_email_all_managers()
    {
        _sender.AssertWasCalled(x => x.Send(Arg

.Matches(y => y.To == "jacob@jonbank.com")));
        _sender.AssertWasCalled(x=>x.Send(Arg

.Matches(y=>y.To =="johnny@jonbank.com")));
    }

    [Test]
    public void should_send_email_to_user()
    {
        _sender.AssertWasCalled(x => x.Send(Arg

.Matches(y => y.To == "james@jonbank.com")));
    }

    [Test]
    public void should_store_leave_request_in_database()
    {
        _emprepo.AssertWasCalled(x=>x.Create(Arg

<LeaveRequest>.Matches

(u=>u == request)));
    }
}

So here we have:

A setup that you need to override and call to setup context specific behavior
small small tests and asserts.
limited setup on mocks, you can use handrolled mocks or the real classes if you prefer (which I do often).
Use AssertWasCalled instead of .Expect() on my mocks

I'll post more examples of this as they come up.

Monorail ActiveRecord and NHibernate

2009-05-21T04:42:00.006-05:00

Ran into this at work yesterday and wanted to save others some aggravation:

Scenario: Need To query a value on a page repeatedly. Another 3rd party program is modifying the data you are querying. (in my case to generate new account numbers, could be anything)

Bad Result: 3rd Party Program changes data in background your page does not update the new value.

So my ActiveRecord query before looked something like so:


ActiveRecordMediator.FindAll(Restrictions//you don't really care do you)

now instead grab the underlying session object:


ISession session =ActiveRecordMediator.GetSessionFactoryHolder().CreateSession(typeof(DpAcct))
var query = session.CreateQuery(//again you dont care)
var list = query.List()
session.Close()

This may not be the best way and my Nhibernate knowledge is rudimentary once you start getting into Sessions, Scopes and Flushing, but it worked to get an ActiveRecord user by.

It seems the cache doesn't know when to refresh when a third party application is involved. This is logical as the caching would be more expensive if it did.

Typemock Isolater and ASP.net Bundle -FREE

2009-05-21T04:35:00.002-05:00

Pretty obvious marketing, but this is an awesome deal so I wanted you my readers in on it too. What kind of irresponsible blogger would I be? So without further ado:

Unit Testing ASP.NET? ASP.NET unit testing has never been this easy.

Typemock is launching a new product for ASP.NET developers – the ASP.NET Bundle - and for the launch will be giving out FREE licenses to bloggers and their readers.

The ASP.NET Bundle is the ultimate ASP.NET unit testing solution, and offers both Typemock Isolator, a unit test tool and Ivonna, the Isolator add-on for ASP.NET unit testing, for a bargain price.

Typemock Isolator is a leading .NET unit testing tool (C# and VB.NET) for many ‘hard to test’ technologies such as SharePoint, ASP.NET, MVC, WCF, WPF, Silverlight and more. Note that for unit testing Silverlight there is an open source Isolator add-on called SilverUnit.

The first 60 bloggers who will blog this text in their blog and tell us about it, will get a Free Isolator ASP.NET Bundle license (Typemock Isolator + Ivonna). If you post this in an ASP.NET dedicated blog, you'll get a license automatically (even if more than 60 submit) during the first week of this announcement.

Also 8 bloggers will get an additional 2 licenses (each) to give away to their readers / friends.

Go ahead, click the following link for more information on how to get your free license.

Hudson, NDepend and OSS - The Good, The Bad, and The Useful

2009-04-26T13:24:00.015-05:00

Patrick Smacchia was kind enough to give me a copy of NDepend so I could give it a good test drive. While we do use NDepend at work, we primarily use it for finding low hanging fruit for refactoring with our legacy software. I decided this would be a wonderful opportunity to see what NDepends story is with OSS and CI.

To facilitate this demo, I used my fresh new BDD Framework Specmaker and Hudson CI server. Normally for CI at work I'd include a copy of whatever software I need in the actual source tree to make it easy to deploy. Obviously, since NDepend is a commercial product and Specmaker is Apache 2.0 license and hosted on Google Code, this was not a viable approach.

So I did the following:

Installed Ndepend on my CI Server
Created a directory to store results
Created a VisualNDepend project. To be able to find my dlls I had to edit the NDependProject.xml file by hand to point to the correct locations. I'd like some nant style Fileset matching instead here. Makes it feel very much like MSBuild.
Setup 2 batch command steps in my build process to run AFTER my normal nant build (seen below, click to view closely) that called ndepend.console.exe and then copied the results to my build folder so they could be archived by Hudson.
Setup the archival rules for my build to grab all of the results that I'd copied to "build\NDependOut" so I could easily reference them after a successful build.

This took quite a bit of fiddling to get working my primary issues were related to Hudson's NAnt plugin (if you reference two nant calls, the first one ends up overwriting your second one so you just do step 1 twice). However once, I got it working my reward was the scolding finger of NDepend. Apparently I have a class teetering on the verge of Cyclomatic Complexity limits (SpecDescription), looks like I have some review to work out some kinks.

Overall Review

The Good - Nice UI for Visual NDepend, GOBS of information, CQL is really well thoughtout dsl for finding what you want so far.
The Bad - Hard Referencing dlls in xml is bad, rule matching is badly needed here. Causes friction when setting up the project for automation. Custom Nant Task it comes with is very limited according to included documents.
The Useful - when combined with CI the comparitive analysis is brilliant. This lets you catch when codebases are declining in quality in an automated fashion. See here for one such example.

Final words, I have to be honest here, I had fallen out of love with NDepend largely, finding it mostly useful only when initially touching a codebase, or when I was first learning SOLID principles. This was of course amazingly useful to my growth, and refactoring efforts, but I was finding less and less need for it with new projects. Looking over Patrick's blog at some of the examples he's come up with using CQL I do not think I've come close to tapping the usefulness possible with this tool. I'll be working more and more towards bending the tool to keep existing codebases in check, and will definitely be using it at home on my OSS work.

Deployment automation and adventures in stupid

2009-04-10T06:26:00.003-05:00

At work we currently do not have a ton of order to software releases and requests. I've worked hard to try to bring more there but it's just not part of the way things have been done in the past and that type of change in particular takes a lot of work and persuasion.

Summary we're better than we were but we're not perfect yet.

I have a confession to make...I was being a hypocrite yesterday. I've been preaching since shortly after I got there "you can automate deployment", "manual deployment is error prone and slow", "integrate regularly, release often". With my background as a sysadmin it didn't take me long to get every at least newer software release auto deployed in some fashion or another. The other push was towards not implementing software in chunks but slowly in front to back layers.

However, about a month and a half ago, me and my co-worker were making changes that would require us to change two products at once. I had stupidly gone off on an unecessarily bold refactoring to modularity of one of those products...so I just let things fall behind and stopped pushing frequent releases.

Month and a half goes by and now the trunk has :

a TON of changes to a variety of processes. Including fixes to things that were major current maintenance time sinks
I had not yet successfully moved all my unit tests around to the different modules so I had managed to eliminate my hard won safety net.
Several database changes that I haven't scripted.
Worst of all code that was behind schedule that had been done for 4 weeks.

So yesterday I finally have had enough of excuses why I wasn't pushing this build out. I manually do everything I had not yet had time to automate, and without all my tests in the proper location!

Finding several stupid kinks, multiple bonehead "I forgot" things, and 15 hours later (with only two hours of sleep) with a process that was supposed to take 30 seconds I had both products released.

I'm quite happy to say now I have an in sync code base, and a big list of fixes to implement which I will release one at a time!

Release Frequently, do not push off things till later, and do things in small small layers.

Update

2009-03-23T06:57:00.003-05:00

It's been awhile since I added anything, but I want to update myself or any future readers on what I've actually been doing with myself the past month.

Got a commiter to Pinsor in Carlos Ble. Very smart fellow and very helpful.
Got plugin support finally worked out for a project at work (but done at home). Will sanitize the code concepts and post here shortly.
Did a presentation of Castle Windsor and IoC at alamocoders that I think went pretty well (need to work on my material and demos a bit, but the talk went well).
Ayende made my head explode with the following post http://ayende.com/Blog/archive/2009/03/20/nhibernate-avoid-identity-generator-when-possible.aspx. Gonna take me a few days to process that one.
Did San Antonio Book Club meet and greet. Was a lot of fun and i think we're going to start working lean into the mix.

Short update I'll try to get more details out of those into another series of posts. I'd like to start putting my windsor material on the blog and hopefully suss out some of the weaker areas.

Bosses and your approach to Agile

2009-02-28T10:18:00.004-06:00

For me the hardest part of Agile by far is getting bosses to understand what it is, why it matters, why what they're doing now is harder than it needs to be, and why what they're doing now is causing the problems they're seeing.

There are a few reasons I've observed and you may or may not run into that I think make this difficult.

Developers tend to talk over "normal" people's heads. I struggle with this and I didn't even own a computer till I was 23, so in theory I should understand the non-technical perspective better. It's just ultimately VERY hard to take a technical and very detailed issue and bring it down to simpler terms and levels.
Bosses that compartmentalize software they buy and software their employees build as entirely different species. If they buy software and it doesn't behave as they expect they'll stop using it, ask for their money back or raise unholy hell. If they spend 50k in man hours for in house work they can sometimes accept it breaking on its very basic designed functionality. I've seen software like that praised because it delivered business value, even if it did so at an excessive unnecessary cost. This ends up you'd think working in our favor, except inevitably when going Agile we have a TON of software to maintain, and none of it is maintainable. This stacks the deck against going agile in so many ways that I'll leave it for a dedicated blog post.
Bosses assume that developers are not able to understand business value. This one has been a huge problem for me. I go out of my way to deliver business value first and foremost, I was a successful manager before I was a software developer. However, generally speaking bosses are correct into thinking this about us a developers, which I'll cover later in another blog post. Needless to say it's blunts our effectiveness at proposing solutions.
Our industry is filled with people that don't deliver what they say. Sometimes it's us just overshooting our goal. Sometimes it's developers who just want to look better than they are and stretch the truth . Finally, sometimes it's people that just are flat out interested in taking the money and running (subtle distinction to others which one of the three types you are). Regardless, some bosses have been burned so much they assume everything is a lie and they've made up their mind when they walk in the room.
Some bosses take a lot of pride in having a big hand in implementation details. Now while this can easily help you once you have agile working if they are interested in helping you produce software that has lots of feedback. However, if your boss is this way, he's probably a former developer in a time far far away, and absolutely convinced about how a software project should be implemented (and assumes that the way they did it then was perfect despite the problems that all projects inevitably have).

There are certainly more, and if anyone wants to share please do. In future blog posts I'll go through these one by one about what problems they cause and hopefully I can work through how to conquer these challenges in the future.

"Do I Care" and "It Depends"

2009-02-18T20:27:00.007-06:00

Between my time talking to Agile Joe when my bank hired him and Jimmy Bogard in the last Virtual Alt.net meeting, I now have two new guiding principles to most development tasks.

Do I Care?

Should you unit test this code? Do I care about that code? Do I care if I change things later and this is not tested?
Should I write this Documentation? Do I care about the docs and what they say? Do I care if I can't remember what I did?
Should I delete this code? Do I care about it anymore? Does it do anything for me now?
Should I follow SOLID principles here? Is the work needed worth it for the payoff?

All of these questions may or may not be yes or no to "Do I Care?". So how do you know if you care or not..this dovetails nicely with "It Depends".

Like it or not one of our primary jobs is too manage tradeoffs. In one case we hamstring ourselves with too much work for little benefit in another we do not do nearly enough "housekeeping work" to make our code manageable.

Full on SOLID on an application you're building for a day and then going to throw away? I'd hope you say "No I Don't Care About SOLID where it causes me more work", an app that will take 2 months to build...if you don't care, make sure it's only the first couple of days and not when it's too late.

Bob Martin focuses on the "last responsible moment" i really always liked that and all but I feel like "Do I Care" maybe appeals in a more plain spoken way. I only follow "good patterns and design" when I start caring, otherwise I fall into the trap of forgetting why i do these things in the first place.

What to work on

2009-02-16T04:14:00.002-06:00

Been quite busy with work for a couple of weeks, but as that is winding down a bit. I think I need another oss project to focus on. Pinsor needs one decent rewrite for clarity, and then some basic feature support and it'll need actual usage to warrant more changes.

Been thinking about what I should work on next?

Job scheduler ( some reason always loved these kind of projects, and could use pinsor on it)
Scrum project manager web app (would have to actually get comfy with pylons/tg/django)
BDD framework in either C#, python or start contribing to spectre in boo.

S.O.L.I.D. Principles in Python

2009-01-31T04:10:00.003-06:00

On the Pinsor google code project I added my Perspective of SOLID principles in Python and their relevance

http://code.google.com/p/pinsor/wiki/SOLID?ts=1233396562&updated=SOLID

Pinsor Python DI/IoC framework

2009-01-26T20:47:00.007-06:00

Well we'll see where this goes, but I finally decided to give back to open source a bit (besides being an advocate and willing trainer for those interested).

Pinsor Google Code Project

There doesn't seem to be much demand for IoC in Python (in fact anger see comments), but there wasn't in the DotNet community for a long time. I believe the trick there was making IoC containers that people wanted to use for their language (read not 4000 lines of xml that you found in Java containers so often back then).

Hopefully, my aim here is to bring auto-dependency resolution and cut down lines of code in an app, everything else that people typically blab on about with IoC/DI is staggeringly less important.

This is an area where I fundamentally disagree with the most users of IoC frameworks who tend to pound DI DI DI and TESTABILTY. Dependency Injection in of itself is a concept that's so bloody simple you can get it in 5 minutes of working with any language if you just accept it (which is by far the hardest part), and not the important part.

The most imporant thing pinsor will aim to bring is less lines of code in moderately complex projects or greater, if it doesn't accomplish that then I'll be the first to delete it and remove it permanently. My focus will therefore be on making sure automatic dependency resolution and convention based configuration are fluid and easy to use.

As I refine the api and get a pythonic configuration DSL up I'll post examples and add to the docs. Till then you may use the unit tests (which are done with a bit of Behavior Driven Development style).

Developer Professionalism and Chad Myers

2009-01-11T12:50:00.005-06:00

Generally speaking I like Chad Myers comments a lot, some of it he has more experience and that leaves me with less material to challenge his ideas, but sometimes he just see's the world the same way I do.

This post is exactly what I've felt for so terribly long, especially the parts about magic, hocus-pocus and charlatans. Its lonnngg past time computing in general move out of the voodoo phase. So here is my advice for aspiring professional developers, technicians and support techs:

Firstly, technology is not to be applied with the same logic as sports teams. Just because someone started liking Windows, Linux, c#, python, oracle, mysql, whatever does not mean that is the only technology in the world and one must support it till their dying day. Things are good/bad on their own merits, and most work in one situation or another, and what was once bad can become good and vice versa.

Let me state this REALLY clearly. Every time you refuse to dive into a technology because "it must be inferior" and you haven't actually tried to get some work done with it, you are hurting YOURSELF.

Secondly, things are not what they seem on initial examination. If you dont know how something works say so! If you don't like something, give it a deeper look and try to use it, talk to those that actually do use it to get work done on a day to day basis.

Thirdly, if something breaks, or acts in a way you didn't expect. Figure out why that is, with analysis and thought. Use the scientific method, only change one variable at a time, and dive deeply until you are satisfied with your findings. Even then don't trust what you've found is gospel.

Finally our job is to serve as tech experts for those that aren't tech experts. This is a delicate job at times, but honesty and building trust is mandatory. Bullcrap solutions/explanations that may fool the customer for a time have a way of sooner or later exposing the bullcrapper as a useless expert to deal with. Regardless, those that give good timely honest solutions and explanations with a touch of humility end up being the ones that everyone relies on.

That's enough soap box for now, but at the end of the day if the industry as a whole acted that way, I'm quite certain we'd have a lot less "awful legacy code", less management interference, and we'd be focusing on delivering business value to customers (and delivering it quickly) instead of geek ego.

Remoting objects (yes I know it's not 2001) without using the MSDN Way

2009-01-02T00:26:00.005-06:00

Ok very few people use Remoting Services any more but it's good in a pinch for quick and dirty way of doing distributed applications.

Now when I went to first use Remoting I struggled to make it fully usable as the books I was using (as well as a quick Google and MSDN lookup) had me all using code like so:

// Create an instance of a channel
 TcpChannel channel = new TcpChannel(8080);
 ChannelServices.RegisterChannel(channel);

 // Register as an available service with the name HelloWorld
 RemotingConfiguration.RegisterWellKnownServiceType(
     typeof(SampleObject),
     "HelloWorld",
     WellKnownObjectMode.SingleCall );

The above code instantiates (calls: new SampleObject() ) an object to be able to accessed remotely..but it cannot take any arguments in the ctor !!

This ended up causing me no end of grief, because if I wanted to write a remoting test, I was pretty much stuck with whatever "new SampleObject()" initialized. Now I was forced to open up configuration changes via setters!

Eventually I scratched the whole thing when I found out Castle Windsor had a remoting facility and just used that to handle my remoting needs and it works quite well.

Looking through Castle Trunk they use RemotingServices like so:


internal static void Marshal(object instance, ComponentModel model)
      {
          MarshalByRefObject mbr = (MarshalByRefObject) instance;
       
          if (!Convert.ToBoolean(model.ExtendedProperties["remoting.published"]))
          {
              string uri = (string) model.ExtendedProperties["remoting.uri"];

             // key logic is here
              RemotingServices.Marshal(mbr, uri, model.Service);
           
              model.ExtendedProperties["remoting.published"] = true;
          }
      }

So with RemotingService.Marshal(object, uri ) is the key here it takes an already intialized class and makes it "remote" and available as a service to another client somewhere else. So much more flexible!

Unit testing Google App Engine

2008-12-29T21:04:00.004-06:00

Was having a massive pain with unit testing with my test development work..well nose plugin and blogging to the rescue.

Will hopefully start making faster progress (code, develop, run test manually, is slower now than tdd).

Google App Engine

2008-12-28T21:20:00.004-06:00

Been playing with this lately. The framework Google provides to access their app engine service is written in python and largely Django from what I can see.

You can supposedly run whatever you want on top of it that uses WSGI with some modifications.

I think I may give this pylons a try at the same time with appengine-monkey . I fear that may be enough steps when using a new framework that idea may be doa but we'll see how much time I have over the next couple of days.

The great dead horse of ORM vs Sproc

2008-12-26T00:54:00.004-06:00

I'm not the first to dive in this at all, but here is my pov. I'm only mentioning this because I still see it cropping up a lot on various blogs and message boards as if there was an actual valid controversy.

Let me recap the basic sides of the argument:

ORM side

This is the haven of coders
Automatically creates your sql for you so you don't have to worry about handcrafting sql
uses normal source code so you get easy refactoring and source control

Sproc side

Position of pretty much all DBA's, some coders typically from a more traditional development software development model.
Been recommended model for development for a couple of decades (since stored procs came into existence )
Optimal performance

Now with those points in place if you were a neutral observer (which is a mythical creature on this subject), you would think i'd come out on the side of sprocs with the benefits in performance and it's proven history...and you'd be wrong.

The above is how I'd say a majority of people I've talked to involved in software see things, but reality is I'd put refactoring and source control as being OVERWHELMINGLY important and performance as being secondary and rarely your limiter.

Finally, I think it's intellectually lazy to just go with what's "been done in the past" regardless of existing realities and what changes today brings.

So lets go blow by blow:

Source Control

Without source control we're left never making changes unless we absolutely have to. This has subtle awful effects on software being reused and growing to match what end users want and need to use.

Source control is one tool that helps software change be safe and thoughtfully done. Sprocs while you can keep the scripts in source control to make or alter the sprocs, this means the dba or whoever is editing them has to be manually disciplined or a third party utility has to make the process seamless.

In my experience this does not work without constantly standing on someones head (including your own) that you always check in sproc changes. In the end, everyone is reluctant to change the application.

Refactoring

So lets say a customer or end user comes to you and requests a simple name change of a column in the database. There may be many reasons for this, sometimes it's for a report, sometimes its compliance with a regulation, sometimes it's someone with power who is whim driven. Regardless you now have several ways of dealing with this.

With the orm approach, you renamed the property in your orm class and since most IDE's have integrated mass rename where all references are updated in one click, all you're left with is renaming the table column (or just mapping the new column name in your orm class).

With sprocs you're in a whole different world. Now there is no auto-renaming tool I'm aware of for sql servers (Redgate may have something expensive) and then I doubt there is one for mysql, mssql, db2, etc. So not only do you to renamed code that you've got to change and test your udf's now that may depend on that column name.

In some cases this could be very quick, but in some practical cases I've worked with this just leads to no one being willing to enact such a change or they just cheat the system a bit and only rename the property or column in one place that makes it appear it was changed.

Things get worse as the issues become more complex however I'll assume you get the idea, again this leads to being reluctant to change the application.

Performance

Performance is nice, but if you REALLY need it, sprocs will not be what makes the difference. I say this with a couple of caveats, depending on your ORM library bulk updates, deletes, and inserts are going to be slow..very slow, but they have to be REALLY bulk. Otherwise scalability in the truest sense is gotten by application design thought through at a systems point of view using messaging systems and growing horizontally.

Lets say sprocs were somehow 25% more performant than hand sql/orm, they're not, but if they were imagine say you had an widget seller app that could process 700 transactions a second.

The widget seller company comes out with a buy one get one free coupon, now transactions are regularly over 100,000 a second. If you were using sprocs or not, you're done for.

However, if you've written an app that uses a messaging protocol and can be spread out over multiple datasources (or even one datasource that's clustered), with several web servers and several middle ware servers...you maybe actually able to scale to that 100,000 a second transactions regardless of if you use sprocs or not.

Scaling out and the use of asynchronous transactions are the actual key way to achieve scalability...everything else is a fake wedge issue.

Most Importantly

Generally speaking anything that gets in the way of giving the customer what they want is very very bad. It leads to applications getting scrapped repeatedly because either the customer has replaced you.

If you're lucky, you've convinced them that yet again they're wrong and did not properly know at the very beginning exactly what they wanted, please deposit another 40k in development to generate the app you desire...this time it'll be perfect we promise.

Heavy use of sprocs are hard to justify when it comes to the maintenance side of things.

Summary

Ultimately I think this does come down to a real valid pull between what dba's have been taught is there job and what agile developers needs are. With the proliferation of agile growing daily (and I'm certain the quality of the agile attempts decreasing as it becomes more and more "the thing to do") it's natural and logical to see this red herring happen over and over again.

Are there other things to consider in this argument? Oh definitely but I've yet to find anything that tops maintainability and adaptability as key reasons to avoid sprocs.

This argument I'm certain will continue indefinitely, and I'll in the near future have to call some 2000 line business logic heavy stored proc that takes 10 minutes to read and then setup on a test database to make sure i'm understanding what my changes will do, but I'll go out of my way to document and then see if i can move it all to unit tested, source controlled, good old boring bog standard code.

Snake Guice

2008-12-21T13:31:00.004-06:00

So I was looking at making my own IoC container in python using inspiration from Ayende's post on rolling your own IoC in a pinch (though I may run into this again when I start playing with ruby) and I came up what I thought would be an ingenious idea to use decorators on methods and classes to do type resolution.
Something like so:


class Foo(object):

@ioc(booable = ghost, mooable = cow)
def __init__(self, booable, moooable):
     self.boo  = boo
     self.moo = moo

I'm sure it wouldn't be popular with your average python dev, but I again really want autowiring of dependencies, and didn't want to make a bunch of factory classes to tie together a bunch of related dependency chains.

Well someone already went with that idea with snake-guice an IoC container inspiried by google guice which is written in java and makes heavy use of annotations the python equivalent of decorators and not concidentally my inspiration for my attempt at an IoC container.

I'll update with more information later as I get to play with snake-guice if I can grok the concepts and code, perhaps I'll dare venture into OSS and see if I can help contribute.

IoC in ruby, python and lack of autowiring

2008-12-14T17:40:00.004-06:00

In my last post I waxed joyously about my love of IoC containers which are third party libs that track all my dependencies.

Now I've been trying to play around with Python and a bit of Ruby lately, and while I just adore the lack of getting decoupling for free (duck-typing gives you this), I enjoy the lack of compiling and I enjoy getting to have a fresh look as a programmer at languages I used to use as an admin for simple scripts.

However, the love fest and productivity ends when I realize I've lost my beloved auto-wiring of dependency management I get with things like Castle Windsor.

Most of the ruby and python IoC containers I found were no longer in active development and were started back when IoC was taking off in Java back in 2004 or so, namely Needle, Copland and py-container.

So I've looked at Spring Python which is active, looks nice and all, but I have to explicitly map components myself. You do get Aspects, service location, and some framework helpers like security, and transaction managment, but without auto-wiring ..... I have to be honest may not be worth it to me at all in a dynamic language.

So this leaves me in a bind. I've been convinced that dynamic languages are the future, staticly compiled languages keep moving that way, but now that i've learned all these awful tricks to make static languages more flexible, and in the meantime found the joy of IoC autowiring....seems like I'm stuck being most productive still in statically compiled languages.

The only alternative I forsee right now if I keep developing in Ruby or Python is build alot of factory classes to tie together large junks of dependency injected code.

Hopefully, I can find an autowiring IoC container for one of my dynamic languages of choice but I dont see how easy it'll be.

IoC and Dependecy Injection

2008-12-14T09:55:00.004-06:00

IoC (Inversion Of Control) is one of many programming concepts that when I was first approached with it I wondered why on earth anyone would have need of such and thing and now I can't imagine a world without it.

The 10,000 foot simple view of it goes something like this:

Anything in your code that requires a resource or code from somewhere else is a "dependency". Examples: database access, another class that provides a method call or service,a web service, etc, etc.
Give all your dependencies an interface or an abstract underclass. have all of your code depend on these instead.
Make all your dependencies come into your class via the constructor, a method parameter, or a setter property. Your end goal is to never have "new" called on anything unless you're working with a class that only contains data . Also using static methods to get around new is cheating and still hard coding a dependency.
Use a third party solution or wire up your own IoC container, that looks at all your dependencies and plays match maker (oh this class needs widget a,b, and c , let me call new widget a, b,c, and add them to this class when called). This replaces all your "Factory" style classes.

Short example of this in code is as follows:


public interface DrawWidget{

void Draw();
}

public class ColorWidget: DrawWidget
{
  public ColorWidget()
{
  //imagine this connects and initializes connection to printer

}
 public void Draw()
{

  // imagine color widget active code here

}
public interface WidgetData
{

  public string[] GetStrings();
}
public class WidgetDb : WidgetData
{
 public WidgetDB()
{
// imagine this sets up connections strings and makes db connection.

}
  public string[] GetStrings()
 {
     //imagine db code here
 }

}

traditional approach:


public class WidgetCombiner
{

public CombinedWidget WidgetPayload()

{

WidgetPayLoad payload = new WidgetPayload();
  payload.Draw = new ColorWidget();//connection to printer must be alive
  payload.Data = new WidgetDb(); //connection to db must be alive

  return payload;

 

}

}

Dependency Injection Approach


public class WidgetCombiner{

private DrawWidget draw;
private WidgetData data;

public WidgetCombiner( DrawWidget draw, WidgetData data)
{
 this.draw = draw;
 this.data = data;
}

public CombinedWidget WidgetPayload()

{
  WidgetPayLoad payload = new WidgetPayload();
  payload.Draw = this.draw;
  payload.Data = this.data;

  return payload;
}

The key difference is WidgetCombiner in the first "traditional" implementation it depends on the printer being live and the database connection functioning. The second implementation does not care at all, it's dependency is dynamic and easily changed to another database, another printer, or just a fake test environment.

YAGNI and when

2008-12-04T13:05:00.003-06:00

I struggle with YAGNI frequently (You Ain't Gonna Need It). But, recently I had some success with just being draconian about. Not adding features unless I could justifiably say "this is adding value X for Y need".

Today, another developer I was working with gave me a bit of a conundrum. We were expecting one of two values from a column in a database (Say "S" and "E"). However, in very very rare cases we could get another couple of different values (Say "X" and "C"). They would likely never come up into our code, and if they did, the rest of the corresponding data would be completely incorrect, and the data would go through the normal paths of incorrect data.

So in this case, should I adjust the code to handle extreme corner cases, that don't effect the actual outcome of the application, but could give some fidelity into logging if an "X" or a "C" actually showed up?

I think YAGNI in this case is telling me not to worry about the C and the X, as they don't actually effect the end result and we have no need for that level of logging fidelity at this time(and changing the where block to exclude the bad possibilities should be enough). But it did give me pause to just leave those corner cases floating out there untouched.

It made me think a bit how in reality a YAGNI style approach probably helped contribute to the justification of y2k problems, and to a degree this was the same issue (which was this other developers contention).

However, I'm pretty certain with modern languages and patterns one could stick to YAGNI and avoid a y2k style mess.

Mako Templates

2008-12-02T23:07:00.003-06:00

Ok so tired of the goofy stuff with Kid even though I did find out a way to debug templates but I was still upset enough that I wanted to go away from an XML based template engine entirely.

Enter Mako. Mako is newish but the preferred engine of Pylons and it works on Turbogears as well. Instead of being xml based it's a run of the mill view engine using python as it's scripting language. This struck me as the easiest and most straight forward for my skill set(especially since I use the pythonic Brail in Monorail).

Turbogears and Mako integration take some hunting. I had to resort to just giving it a stab how I wanted it to work, googling my errors, and then bit by bit piecing together the sections of configuration i needed.

Finally I groked the concept thanks to Wesley Chun the author of Core Python Programming who'd run into a similar conundrum (at least it makes me feel less stupid) .

Key excerpt here:

2. another problem that people have is that in your controller, your @expose decorator call tends to have more of a "full-path" to the template when porting from Kid, i.e.,

@expose(template="tghello.templates.hello") def hello(self): :

when switching to Mako, because of your mako.directories list, you no longer have to give the entire path, meaning that the failures are happening because it's trying to find that full name as a template file name, which isn't going to work.

Finally, played with this briefly feel much more at home with this templating style.

Castle Active Record and ActiveRecordMediator

2008-12-02T20:29:00.004-06:00

the ActiveRecord pattern as it's typically implemented works like so.
step 1 - Take a base class with all the persistence methods you need (save, update, delete, find, etc)


public class ActiveRecordBase{

public static object Find(ICriteria criteria)
{
//return object

}
public void Save()
{
//save object to db here
}
// rest hidden
}

step 2 - take all your db classes and have them inherit from them (optionally passing on column configuration or whatever you'll make available).


public class Account : ActiveRecordBase
{

public int Id{get;set;}
public string Name{get;set;}
public AcctType Type {get;set;}

}

client code ends up looking like so



public void static Main(string[] args)
{
Account acct = new Account();
acct.Name = "Max Earnings";
acct.AcctType = AcctType.Checking;
acct.Save()

Account[] acctsfromdb = Account.FindAll();


}

Now alot of people like this pattern. They'll attach behavior or methods to the properties or even add their services to the same class (in the above example maybe adding a method that calculates current interest rate against a lookup table).

I however have always had issues with unit testing this way. You end up testing against a database directly (which is substantially slower), and your class becomes decorated with all these persistence methods. Plus I've never personally enjoyed mixing data classes and method classes (called services in Domain Driven Design), this confuses the picture for me and muddies things up (what do you do when you have an interaction between multiple classes for example?)

So how do you get the simplicity of db access with ActiveRecord but not have to worry about having database tied classes? Well you can use an orm that's persistent ignorant like Nhibernate (or several others), which involves alot of by hand configuration (I say this watching the fluent nhibernate project closely), session, transaction handling, and generally gives you more control than you need for alot of starting projects. If only you could find a nice middle ground.

Enter Castle ActiveRecord and ActiveRecordMediator class. ActiveRecordMediator is NHibernate for dummies (or lazys). So back to our earlier example


[ActiveRecord]
public class Account
{
[Property]
public int Id{get;set;}

[Property]
public string Name{get;set;}

[Property]
public AcctType Type {get;set;}


}

now client code looks like so


public static void Main(string[] args){


Account acct = new Account();
acct.Name = "Max Earnings";
acct.AcctType = AcctType.Checking;
ActiveRecordMediator<account>.Save(acct)

Account[] acctsfromdb = ActiveRecordMediator<account>.FindAll();


}

So recap the difference in code is small. Lines of code is about the same..but now I can decouple the use of my objects away from database...but HOW you ask?

Ahh yes let me carry you that last step.


public interface IRepository<T>{

public void Save(T t);
public T[] FindAll();
public void Update(T t);
//etc etc


}
public class DbRepsository<T>:IRepository<T>where T : class
{

public void Save(T t){

ActiveRecordMediator<T>.Save(t);
}

public T[] FindAll()
{
 var records = ActiveRecordMediator<T>.FindAll();
 return records;
}
//etc, etc

}
public class Client{
private IRepository<Account> _db;


public Client(IRepository<Account>() db)
{
_db = db

}


public void AddAccount(Account acct)
{
_db.Save(acct);
}

}

public class FakeDb<T>:IRepository<T> where T : class
{
private List<T> records = new List<T>();

public void Save(T t){
 records.Add(t);
}
public T[] FindAll(){
return records.ToArray();
}
//etc etc

}


[TestFixture]
public class TestClientCode{

[Test]
public void should_save_record_to_repository(){

var memdb = new FakeDb<Account>();
var client = new Client(memdb);
Account acct = new Account();
acct.Name = "Max Earnings";
acct.AcctType = AcctType.Checking;

client.Save(acct);

var fromdb = memdb.FindAll();
Assert.AreEqual(1,fromdb.Length);

}

Final Recap if you look at the test we can now easily write unit tests (with no db to worry about). With the simple example provided here there isn't much obvious gain, but start adding your persistence classes to controllers, services and any other usage you can think of and this pays back the initial work we did here 10 fold.

Issues with Kid templating

2008-12-01T21:44:00.004-06:00

Was working on my little Turbogears app I'm building and ran into some issues with the Kid template system that's default with turbogears 1.0.

Now I've wasted several hours just messing around with master and layout files. I can build my own master file and have it work quite well, but no one wants to see my websites so I grabbed a template off OSWD and it was even in strict xhtml.

First, I try the master approach. This is to me a pretty noisy process (have to specify match parameters in html, head, and what i want to be the content area, i learned through trial and error this does not seem to be optional) but at least it looks like a complete xhtml file to a web designer.

Now the specific pages just need a py:extends="'sitemaster'" attribute in the html tag and boom....it blows up with an odd error (

Template file "'sitemaster.kid'" not found
while processing layout="'sitemaster.kid'"

.
)

I dive into the kid language specification and find the py:layout="'sitemaster'" option. this looks much better. far less noise and at least somewhat closer to what you'd find in NVelocity or Brail.
However in the end once I've made the same changes, done a test case that works with some hand thrown together html. When trying to load up my 3rd party template it blows up with same odd error.

Long story short, I narrowed enough down to find there is something in an WC3 xhtml strict compliant doc that throws the kid parser for a loop. I can delete all the html in the body and it works.

I believe the way kid ends up approaching things is ultimately to blame here, and I'd have to change the template output to strict xhtml or something to get this all to work..but ultimately I shouldn't have to think about this at all.

It is my belief that view engines should stay out of the way of the html entirely, they shouldnt barf on it, and if they do they should tell you what element is causing you the problem, or where the parser gives up at the horrible mess you've sent it.

I get that Kid is trying really hard to make a designer friendly view engine, but in this instance if I were working with a designer and we got this error when he made an html change..and he actually did it 3 days ago, and he did a bunch more changes since...we'd be struggling for hours to figure out what was the offending piece of html he added.

Now I'm sure experienced users of Kid would chastise me, and that I'm supposed to learn all the ins and outs and then it'll all be worth it (it may very well be, but I'm not convinced yet).

I had a look at genshi and it looks like the same model but better in some ways (extension is .html..this alone makes it worthwhile for dealing with skiddish designers). Maybe it'll tell me where this html blows up...regardless I'll say it again..view engines should not care what html I throw it.

Turbogears, Pylons, and Django

2008-11-30T13:46:00.004-06:00

Been wanting to do some high productivity oss work at home lately. At first I was gonna write something in rails, but being as I have a bit more experience in python that Ruby, I decided to give the Python MVC frameworks a look.

From what I can see so far from my research and a few hours coding in django and turbogears.

TurboGears

Middle of changeover in orm technologies preferences. Version 1.0 prefers SQLObject, but has support for SQLAlchemy. 1.1 Beta has tigher integration with SQLAlchemy.
Horrid name, but seems like more classic MVC framework, has some convention of configuration out of the box.

Django

Has great docs. Can find loads of tutorials for it.
When I used it seemed I had to do a lot of manual url configuration. I found this obnoxious.
Nice out of the box admin pages, that are easy to set up. Some people find this gimmicky but I had little issue with it.

Pylons

Have not had time to code with this but from the docs appears to be a pythonic rails (claims as such).
It's a bit new and docs struck me as a bit more lacking.
Has conflict with turbogears somehow. Not enough of a python guru to figure this out. Other people had same issue, had to run one or other on the install however.

Overall my impression of the python MVC trio here was solid. It still does seem rails has convention over configuration down better, but I haven't used anything yet in any language that was as well thought out for "good defaults". Turbogears in particular is still better than say Castle Monorail, but that has more to do with the differences between static compliation and dynamic (look at the differences in the Castle's Active Record and Rail's to see what I'm referring too).

Why oh why streamreader

2008-10-23T20:56:00.008-05:00

While I was going through some playing around with python i was really impressed with the following bit of functional joy.


inList = open(fileLoc, 'rU').readlines();

I was so happy to find that little tidbit and sad to not have used such a thing in C# ever. I always used the awful:


List<String> strings = new List<String>();
using(StreamReader reader = new StreamReader(fileloc))
{
while(line = reader.ReadLine()) != null
{
strings.Add(line);
}
}

This is the type of code you see in MSDN and in various "Professional C# 2.0" type books, they'll even show you TextReader to boot. However for the vast majority of cases the least bug free and easy to use appears to be something I totally missed for the past couple of years:


string[] readText = File.ReadAllLines(fileloc);

This has been around since .net 2.0, opens the file, reads it and closes it. At this point I'd argue that the shorter option should be shown primarily to newer students and the remaining features left in an appendix somewhere.

EDIT: Jon Skeet pointed out his simple class for doing the same above but with better performance. http://csharpindepth.com/ViewChapterNotes.aspx?Chapter=6

AOP: what is it and why do I care

2008-09-27T08:00:00.011-05:00

Aspect Oriented Programming or AOP. This is Yet Another Acronym (or YAA ) but for most people and most cases it's really just another recipe to solve problems in your programmer cookbook. There are many resources out there on AOP on the web and I won't attempt to cover all details of AOP.

For those new to the concept, AOP will be most clear to you through the lens of logging concerns. How many of you have written or seen code like the following?:

DoSomething action= new DoSomething();
string result = action.Execute();
Logger.Log("Info", result);

Or even worse:

DoSomething action = new DoSomething();
Customer result = action.Execute();
LogCustomer.Log(result.FullName, result.Address, result.Phone);

Only you see it every 3 lines and scattered everywhere, throughout the code base. Do something as simple as changing logging framework, changing what you log or adjusting your static logging class can end up being a nightmare. AOP attempts to solve this more cleverly by using a combination of rules called "pointcuts" to match what you log, and then executing functions called "advices" .

The flexibility and implementation of pointcuts vary greatly from one AOP framework to another, so you'll have to consult your framework of choice's docs. But I can give you an example using IoC container Castle Windsor:

1: WindsorContainer container  = new WindsorContainter();
2: container.Register(AllTypes.Of.FromAssembly
(Assembly.GetExecutingAssembly()).Interceptors.Add(new InterceptorReference(typeof(LoggingAdvice))).Last));

Now lets recap what's going on here? First line initializes the container, second line registers all classes that implement the ICommand interface in the currently running assembly, and then adds the LoggingAdvice. You can change this to all classes, or only individual classes but that's an exercise I leave up to the reader. Lets show the code now:

public interface ICommand{
string Execute(string sendString);

}

public class Command: ICommand
public string Execute(string sendString)
{
   string toconsole  = sendString + " this is from the console";
   Console.WriteLine( toconsole);
   string retvalue = sendString + " this is returned by the method";
   return retvale;
}

This is a simple enough class. It writes the sent string to the console with some additional information, then returns the sent string with where it is in the Execute Method.

And the LoggingAdvice is as follows:

public class LoggingAdvice:IInterceptor
{

public void Intercept(IInvocation invocation)

  {
              invocation.Proceed();
              Object argument = invocation.GetArgumentValue();
              Console.WriteLine(argument.ToString() + " from LoggingAdvice" );
              Object returnValue = invocation.ReturnValue;
              Console.WriteLine(returnvalue.ToString() + " from LoggingAdvice" );

   }


}

note the invocation.Proceed() method call. This ends up having the "Advised" or "Intercepted" method execute then and there. Then the invocation will be able to get the return value. This means you could move the proceed call to just before the ReturnValue call.

Now client code would look something like so:

class Program{

static void Main(string[] args)

 {
   WindsorContainer container = new WindsorContainer();
   //add registration code from above here
  ICommand cmd = container.Resolve();
  cmd.Execute(args[0]);
}

What's going on here? Windsor retrieves an instance of ICommand which will give you the Command class. The Execute is called on the ICommand instance with the args from the command line passed in. You'll find a result similiar to so when run from the command prompt:

1: program.exe "doing stuff"
2: doing stuff this is from console
3: doing stuff from Logging Advice
4: doing stuff this is returned from the method from Logging Advice

Let's recap, our core code is making no direct calls to a logging class and it is not sprinkled all over our code. While in this trivial example the gains are not realized but anything larger than 300 lines of code we're going to see a benefit from using an AOP framework.

In upcoming posts I'll cover other frameworks and more common uses of AOP.

Quartz.Net and Castle.Scheduler for scheduled jobs

2008-09-23T22:35:00.005-05:00

I've had the opportunity to work with both scheduler frameworks in the same project (ended up switching out Quartz.Net for Castle.Scheduler) and here is what I found.

Quartz.Net

Positives:

CronTrigger alone is almost worth it. Anyone with a unix background (like me) finds this a big win.
Based on a very mature Java based project so in theory it's design is well tested.
Has support for supplying different Calendars (think holiday calendar, being loaded into a "job" so that it knows not to fire on your organizations particular holidays, this is nice to have in banking for example).
comes with command line client and service. This wasn't out when I was using it and I had to implement my own versions.

Negatives:

Complete Seperation of Triggers (scheduling objects) and a "JobDetail" (think name, description, etc). This in my experience lead to alot of confusion with developers and caused alot of inconsistency. Example: When wanting to reschedule a job, you had to reschedule it by it's trigger name. Which would cause problems if you had that trigger setup on more than one "JobDetail", and also would require you to find the trigger name of that job anyway.
A lot of string literals. Just my personal take don't care for this.
Some incomplete implementations. Example: when using 0.9.1 I had a dev who was stuck on why NextFireTime wasn't showing up on a Trigger. Some things are just tagged as not implemented yet.
MegaClasses. Lots of methods, lots of noise. Again this is a port so it's not like Quartz.Net can vary too terribly much from Quartz.
Only one developer that I can see. He works hard, pushes out releases frequently, but the Castle team is top notch and has many qualified contributors, it's not a fair comparison.
RamJobStore was solid, ADOJobstore was a bit flakey and didn't work nearly as well.

Castle Scheduler

Positives:

Leverages Castle Windsor. If you use it already this is huge plus.
Simple small easily understandable classes.
JobSpec easy to understand and transparent to the junior devs. Includes a trigger property so that there is only a 1 to 1 mapping between triggers and jobs.
SqlJob store is well supported and works about as well as the InMemoryJobStore.
Out of the box clustering of schedulers using a db backend to track jobs between the schedulers.

Negatives:

As most castle projects, may not be enough docs for less experienced devs unused to reading source code and unit tests.
Limited out of the box trigger support. You'll have to implement your own triggers to get something outside of daily and interval executions.
You have to be comfortable builing Castle Trunk. This is a limiter to new users obviously.

Summary
.
Both of them do the job if you need a job scheduler. If you already "speak Castle" and are comfortable with that tool chain then Castle.Scheduler is a much lower friction option. If you don't already use Castle (which I love btw) , or you need complex scheduling then Quartz.Net may be more for you. However, fear the AdoJobStore and prepare to use another method to persist you jobs, if you find friction using it for higher performance scenarios.

I'll try to make time showing the basic use of Castle.Scheduler not already covered in docs later in the week.