Thursday, July 15, 2010

Anti-Pattern: Too much of your application is about interacting with external resources

Firstly, what am I talking about? Applications that meet some of the following descriptions:
  1. Stored procedures with a fair amount of conditional logic or complicated business rules buried in a sub-query (some would argue sprocs at all).
  2. Web pages with lots of conditional output again encoding business rules in snippets of view logic. if else checks that display the latest price for something if before and certain date or a default price.
  3. The majority of ‘unit tests’ require a database, or web and application server to be up and running.
  4. The majority of code files make reference to language default I/O or database libraries.
  5. Hours are spent trying to determine if differences in version of infrastructure are the cause of certain bugs.
  6. Changing database schema results in hours of refactoring, as hundreds of querys and sprocs are hunted through to see if this index or that index is being properly hit.
Why is this an anti-pattern? First before I bore you with endless reasoning behind SOLID principles, proper OO etc if you’re reading this and you’re of a bent that disagrees with me so far entirely..none of that will mean anything to you, and none of it will mean anything to your customers so let me put this in the most practical terms I can.
It is not testable in the slightest in any practical sense. You will be counter with well that with that sort of application and making everything a front to back test that you KNOW when things are working, yes but you RARELY if EVER know in a quick sense why things are not working. If i can reliably eliminate any chance that my actual code or business logic is the cause of a problem, if all of my fancy business rules are in something that can be run thousands of times a second, then I can throw all sorts of corner cases at my code base and not increase my verification time. If the majority of my application is business logic and plain old code then I can save my slow manual testing to the areas that are so bullet proof and only have a slice of my application doing front to back testing to yes, make sure things really work.
Note: Cross posted from Polyglots R Us.

Permalink

Saturday, July 10, 2010

Project Management in Java: A Confused .NET Developer’s Perspective

When I was first introduced to workplace Java the amount of ways one could define a project appeared to be restrictive, confusing and a point of frequent friction. While those things may all be true, it’s a great deal better than I would have expected since I was coming from a world where an upgrade to a newer version of Visual Studio requires your whole team to sync up to that same version!  So how do cross IDEA teams get any work done in the Java workplace? Do Java OSS projects require everyone uses Eclipse, NetBeans, or IntelliJ?  Does every new IDE need to have import/export filters for all of the other existing ones and if it becomes popular do the already present IDE’s have to include this? Which system is the best and what matters?
To answer these questions let break down what makes a project and while doing so compare and contrast the Java perspective with the .NET one.

Compilation

Java IDE’s  (Intellij/Eclipse/Netbeans) – basically run “make” on the fly and pass everything to javac.
JAVA IDE-agnostic project systems  (Maven, Gradle) – adding all the references contained directly to the classpath and then calling javac.
.NET (Visual Studio,MSBuild)  – MSBuild feeds all references to csc.
In summary, there is more similar than different here.

Strong Versioning

Java IDE’s – at least in the case of IntelliJ I ended up manually writing a MANIFEST.MF file.
Java IDE-agnostic project systems – still manual editing a lot of the time but usually just one line already filled out by your template that you edit.  I know NetBeans gives you a GUI to edit this.
.NET – GUI but you can manually edit an assembly.cs file.
In summary,  Java IDE’s at least in the case of IntelliJ really are using default Java by hand stuff.

Dependencies

Java IDE’s – All use convention! Basically, search for jar files in your source tree, if the IDE finds them it assumes you need them and adds them as references in your project format. Works pretty nice on a whole and allows for manual tweaking in the oddball cases.
Java IDE-agnostic project systems – requires explicit dependency reference by version, however they will download them automatically from remote locations and put them in your file-system in a specific way (defined by them unless overridden by you). Since both of these are also responsible for your compilation you have to basically run everything through them or your IDE has to be aware of them.
.NET – requires explicit reference to DLL’s on the file system.
In summary, .NET and the Java IDE-agnostic systems are VERY similar here in approach, whereas the Java IDE’s take a convention over configuration approach. I’d say there are times I definitely prefer the Java IDE approach and there are times the capabilities of the Java IDE-agnostic systems are worth it.

Source References

All Java solutions – do reference by directory structure and wild card matching.
.NET – explicit references to each class file .
In summary, this is the HUGE difference. Java takes more of a Make based approach where it’s EXPECTED this will not be the only IDE/project system ever used on this source tree, and this is where the Visual Studio/.NET experience starts to fall apart when trying to use cross-ide’s or versions. It has forced everyone to implement and keep up with MSBuild’s format to enable other IDE’s to be used, and don’t talk about actual alternatives to MSBuild. At this point even though things like NAnt and Rake/Albacore are more popular among agile .NET teams than scripting MSBuild almost all of those scripts still call MSBuild to do the compile step!

Questions Answered

Q. How do Java OSS projects deal with all the different IDE’s out there and even different project formats?/How do cross IDE teams get work done?
A. They don’t have to worry about it as much as .NET devs. A friend of mine who a lead contributor to the Apache Cassandra project in fact included a Maven file for the “enterprise types” but by and large use Ant’s to do their builds and he himself has used a number of IDE’s on that same codebase. .NET OSS development is still a bit hamstrung by MSBuild and really alternative IDE’s are keeping in sync with that project management format. There are real problems with style and code format guidelines, but there are usually workarounds or ways of solving this if code style guidelines is something you really care about ( which I don’t).
Q. Which system is the best?
A. There is no one size fits all answer here. I now know Apache Maven solves some problems for people and even though I don’t appreciate it’s approach to things, it’s done some valuable good for the community.  The IDE-centric approach really works pretty well a lot of the time for projects. Gradle, while being the closest to what I want is very young and has limited IDE support, at least it’s not easy enough for this Java n00b to figure out how to use with IntelliJ without some friction.

Summary

I hope that all wasn’t too confusing, it sure was for me to go through while I had all these options , and in a completely different environment to what I was used to. Worse a lot of the folks I talked to saw these IDE-agnostic project systems as solving problems they didn’t have. Maven, Ivy, etc was a world that was buzzword laden (I’m really looking at you Maven and Ivy), involved heavy setup time for relatively simple tasks, incomplete documentation (looking at you Gradle) and was frequently badly misused (those who know me know what I’m referring too here). In the end an IDE-centric approach in the Java world does not seem to tie you to that IDE in any way I can find, and for getting started had little friction, and for people new to Java it seemed to be the best way to get started. Besides in Java at least it’s easy to change your mind later.
Note: Cross posted from Polyglots R Us.

Permalink

Thursday, July 1, 2010

MySQL 5 Performance Tuning Toolkit

Recently we’d played with table partitioning and because of the limitations of it and some decisions we’d made a very long time ago we ended up spending a couple of days tracking down hotspots. In the process I picked up a few tricks, I’m not even an authority on any of this but I figured it’d help out others in the same spot.

Logging Toolkit

If you have a slow loading page or test and have no idea which one of the thirty or so queries going on is the cause how to do you find it.  Slow query log and logging queries that are not using indexes. Adding the following to your my.ini, my.cnf or whatever your mysql config file is named on your system and you’ll be able to get a list of offenders to further investigate.

MySQL Console Toolkit

SHOW INNODB STATUS – gives tons of useful information on what the current state of deadlocks, transactions and all sorts of information.
SHOW FULL PROCESSLIST – shows the current threads and queries running on them. running this over and over again can show you queries that are hanging.
EXPLAIN <QUERY>– Just in place of query type your full query statement, particularly in sub-queries, this is great for pointing out potential problems, in my most recent case I had an uncacheable sub-query, pointing out quickly something that was just not going to work no matter what.

Final Notes

Anyway this is only a smidgen of the useful stuff you need in your toolkit and I’m certainly no expert. I want to plug High Performance MySQL: Optimization, Backups, Replication, and More and the http://www.mysqlperformanceblog.com/ for it’s help and as a reference for those wanting to go into deeper study.
Note: Cross posted from Polyglots R Us.

Permalink