Iterate Me: Brian Klaas: Blog - Software Development

Book Review: Spring Recipes

Tue, 07 Jul 2009 13:29:00-0400

I'm a really big fan of ColdSpring. I'd say that it is one of the key enablers for enterprise-class, object-oriented software development using ColdFusion. I can't imagine building a CF app without it now. ColdSpring took its inspiration from the much larger, much more powerful Spring framework. ColdSpring represents but a mere subset of everything that Spring can do. That's not a knock against ColdSpring — Chris Scott was very clear about his intentions when he originally developed the framework. As I've tried to expand my knowledge of software development, and, more specifically, Groovy and Hibernate in particular, the Spring framework has come up again and again. I decided it was time to move beyond my ColdSpring-based understanding of Spring and try to learn about Spring itself. The first stop on this journey has been "Spring Recipies: A Problem-Solution Approach," by Gary Mak. This is an intensive, though not exhaustive, look at what Spring has to offer. It covers all of the major components of Spring, above and beyond its very robust Dependency Injection framework. The book covers numerous aspects of the larger Spring framework, including:

The Basic and Advanced Spring Dependency Injection Container
AoP in Spring: Classic, 2.0x and the AspectJ
Spring JDBC templating
Spring Transactions Framework (for database transactions)
Spring + Hibernate
Spring MVC
Spring Testing Support for JUnit and TestNG
Spring Security
Integrating Spring with Portals and Portlets
Spring Web Flow (of particular interest to me)
Spring "Remoting"
Spring support for scheduling and email

There are a few other topics covered as well, but the point of listing all that is covered in the book is to give you an idea of the depth and complexity of the Spring framework. Dependency injection is just one part of this ginormous (and some say needlessly complex) framework. One of the nice things about Spring is that although it's large and complex, you can use just the pieces you want: you don't have to utilize the _entire_ framework in building your Java application. Each one of the above topics is given fairly good coverage in the book. Some topics are relatively straightforward and so don't get a lot of pages, while the more complex topics get many more pages. The writing is clear and the examples quite good, even for someone like me who isn't exactly an expert in Java development. It was very clear to me, after completing each major section, why I would or would not want to use a particular feature of the larger Spring framework. The examples, which are numerous, demonstrate Spring's power at templating otherwise duplicate code throughout whatever application development needs you may have. Yes, this results in more and more XML, though annotations go a long way to reducing the amount of XML in the configuration files. One flaw in the book — and it's more a limitation of the printing industry than anything else — is that once the basic examples are out of the way, the author refers you to the online documentation for additional examples or much of an in-depth coverage of a particular aspect of the framework. This isn't entirely surprising, though. There are many books written about the individual parts of the larger Spring framework, so it's doubtful that one, single book could cover everything in depth and detail. The book's clarity and intelligence has inspired me to learn more about some of what Spring can do. Spring Web Flow is really interesting to me, as a developer of Web applications, and I plan on learning more about it and figuring out how I can implement some adaptation of it in my own applications. I suppose that I should go and try to build a Spring-based application as well, but I like to bring things back to ColdFusion whenever possible.

Be Part of the Solution

Thu, 30 Apr 2009 11:44:00-0400

A blog post over on Vitamin (a fine resource for Web application development articles) got me reading a commentary by Virginia Hefferman at the New York Times Magazine about the knee-jerk reflexivism so rampant in online "commenting" tools. It's a worthy read (even if she fails to make the distinction between generalist sites (ie; newspapers) and specific, targeted communities). It got me thinking, though, about something I'm making a conscious effort to do in my everyday Web application development work: be part of the solution to a problem and encourage others to be the same. Here's the crux of the matter: You can't just say that a Web application or framework or library is terrible. You have to be willing to say how to make it better. It's easy to dump in 140 characters or less, but how about you try to figure out a way to make it better? Here's a classic example of this, refracted through the lens of my day job:

Faculty member: This is the worst application I've ever had to deal with. Me: If you have suggestions on improving or fixing the application, I'd love to hear them. Faculty member: [Silence]

Now this may just be a case of a user feeling that they have vastly more important things to do than figure out how to solve someone else's problems, or that we should be good enough at what we do to just get it right the first time out. They'd be partly right on both fronts. But it's so easy, so trivial to dump all over the work of others, particularly when enabled with the commenting power found in blogs, Facebook, Twitter, and elsewhere, that we forget that if we know how to do it better, we should ourselves up to the same possibility of negative attack and offer how to do it better. I'm not saying that you then have to go out and implement every feature request or idea someone comes up with (and a single feature request should rarely turn in to an implemented feature), but you can — should — demand that they who complain be part of the solution. I'd argue that the same should go for movies, literature, and politics, but that's a whole other post.

JK Rowling on the Power of Failure and Imagination

Mon, 06 Apr 2009 09:44:00-0400

I tweeted about this earlier today, but JK Rowling's 2008 Harvard commencement speech on the twin pillars of failure and imagination was an awfully good read (though not good enough to topple Steve Jobs' Stanford commencement address as the best I've read). The speech was timely for me as I've recently been dealing with the issue of failure in relation to a product we rolled out at the beginning of the year which was, ultimately, deeply flawed and brought upon my head the wrath of many, many users. The product didn't fail to meet the feature requests and needs of the users. It had all the features they had requested and then some. Where it failed, miserably, was in implementation. From the overall UI direction, to security, to QA testing, there were failures large and small which, when driven home through extensive use of the product, made many in our user base not just dislike the product, but hate it. I've now come to look at it as the Word 6.0 of our product development history. There's a silver lining, though. Through failure, we've gone hat-in-hand to our users to ask how we can do things better and have gotten some really great ideas on how to fix things in return. We've taken a look at our overall development process and tweaked it to improve visibility earlier in the product development cycle. We've reviewed QA and testing practices and empowered that team to speak up earlier and hold releases if evaluative criteria have been met. Failure is a powerful tool in the software development arsenal. It's scary, sure, but a superb teaching tool because it forces you to imagine how to do things better and then to actually do better the next time around.

A Small SVN/Eclipse Pointer

Mon, 09 Feb 2009 13:31:00-0400

I'm sure that this is well documented and common practice, but I'd never run in to a problem submitting my Eclipse .project files to a Subversion repository before. I mean, why not, right? The .project file is included when you first try to commit, it's in the same folder as the rest of the files versioned by Subversion, so what could possibly be wrong with doing that? When I went to check out a project using Subclipse in to a fresh install of Eclipse, I found out why. The new project wizard won't let you check out a Subversion-controlled project if there's a .project file in the contents you are trying to check out. Oops! The moral of the story: don't put your .project file in to the repository. Not that it needs to be there anyway, but that was my noob mistake. As I couldn't delete the .project file using Subclipse, I fired up the lovely and handy Versions to take care of that for me. Once the .project file was deleted from the repository (along with a note about my foolishness in the commit log), I was good to check out using the new project wizard in Eclipse.

IE Caching Wastes Hours of My Life

Tue, 30 Dec 2008 08:45:00-0400

I should have known better. I should have remembered. I've seen it before, but it bit me again: IE's abusive caching destroys the quiet peace of developing with JavaScript libraries like Prototype. Here's the issue: I was updating the content of a <div> on a page via a form field on the same page. Very simple stuff. It worked flawlessly on Safari and Firefox (no surprises there), but failed on IE7. I tried the same code on a different machine running the exact same version of IE7 down to the xxx.xxx.xx.xxxx build, and it worked just fine. I tried it on a third machine and it failed. WTF? The culprit, as I should have remembered from reading endless blog posts every day, was IE's overly aggressive caching strategy. Now on the Web application in question, we're setting HTTP headers to "No-Cache" to avoid these kinds of issues, and that works well for full page reloads. It does not, however, always work for XMLHttpRequest calls. The default setting for IE is to "Automatically" check for new versions of Web pages when requests are made. If you look under "Tools" -> "Internet Options" -> "Browsing History" -> "Settings," the default is to check for new versions of Web pages "Automatically." However, IE's idea of "Automatically" doesn't take in to account XMLHttpRequests. The solution is simple: append a random value to each AJAX request and the problem vanishes because now each URL being requested is "unique" and IE's default, aggressive, abusive caching strategy is foiled. JavaScript example:


var cfcURL = 'myService.cfc';
var randomizer = Math.floor(Math.random()*50000);
var params = 'method=getContent&differentiator=' + differentiator;	
var myAjax = new Ajax.Request(cfcURL, {method:'get',parameters:params,onComplete:udpateSomeDiv});

Hours of my life wasted on something that should just work. Oh, that's right, that's Apple's motto, not Microsoft's.

How Do You Stop Users from Creating Duplicate Accounts?

Fri, 24 Oct 2008 21:47:00-0400

I'm prepping to build out a consolidated registration site for a couple of the services that we provide at my place of work. Instead of replicating a registration process across a couple systems, we're building a centralized registration system and users can then take those credentials and log in to the services that they need. (We're not replacing existing user accounts with OpenID and OAuth or anything like that — sorry!) There's a lot of work we'll be doing to make the process simpler and more elegant than the current set of processes. One major hurdle remains, however: how to prevent users from creating duplicate accounts. When users register for a site — any site — and if they don't access the site right away or within the next couple days, they often forget their account information. If they wait long enough, they'll even forget the email address with which they registered. This is particularly problematic in online education, as someone may take a class, then not take another for a year or two, but return to take another class at some distant point in the future when their email (or physical) address may have changed. We don't want them to make new accounts because then we don't have a unified history of their activity (and this can cause problems with things like prerequisite courses). A common approach I've seen and we've used in the past is to check against the provided email address. If there's already an account in the system with the given email address, then you let the user know that they already have an account and provide them a link to retrieve that information. However, if the user is trying to register with a different email address, they're going to make a duplicate account. The system isn't going to know that bsimpson@hotmail.com and santoslhalper@gmail.com are the same person. So what to do? I'm considering the following, but am very open to suggestions:

Check the first and last name: if there's a match to the first and last name, suggest display a list of email addresses that match and say "Hey, if one of these is your email address, go ahead and retrieve your account info."
Check the first and last name and city and state or country: It's very unlikely that there are two Janet Halzipools in Buffalo, New York, but one never knows. (In parts of South India, for example, a name like Omar Khan is very common, though less so in Reno, Nevada so maybe this approach won't work particularly well.)
Check another unique identifier: We don't/won't/never will store Social Security Numbers, but if you have another unique identifier, you can always check against that. The University that I work for distributes unique identifiers to some, but not all, of our user base (hence the need for a separate set of authentication data). This would work pretty well, but if the user no longer has access to the email address which is associated in the system with this unique identifier, how do they retrieve their account information? Do you ask them to engage in the manual task of contacting support and then waiting for a response from support to get information updated in the system?

Once of my key considerations about any of the above strategies is this: if you find what you think is a match, how do you prompt the user to see if this is them without making them think their account security is at risk? Again, any suggestions are welcome. Even if I can't remove duplicate account creation, if I can significantly reduce it with one or more of the above strategies, I'll be happy.

Considering Mate and Swiz

Wed, 01 Oct 2008 08:39:00-0400

I have a confession: I don't get to do much Flex development. I read up on it. I've attended trainings. I go to classes. I watch webcasts and screencasts. I don't, however, have a lot of opportunities (or time) to build Flex apps. A lot of what I do mixes rich media and lots of text, and I'm not sure that Flex (or Flash) is the best environment for lots of text. Feel free to argue or disagree in comments below. One thing I do know is that I don't like to work without frameworks, regardless of the language in which I'm working. Frameworks and standard libraries can take a lot of work out of developing applications, and can often help you organize your application according to best practices. In ColdFusion, I don't build applications without Mach-II (or Model-Glue, but I'm mostly a Mach-II developer). In my Flex development experience thus far, I've been working without additional frameworks (because Flex is a framework, after all). I've played with Cairngorm and understand how applications need to be laid out using that micro-architecture. It's a bit unwieldily for me. PureMVC is powerful as well, but, again, requires all sorts of extra classes and overhead that I'm not a fan of. (And, honestly, I'm not making AS3 apps, I'm making Flex apps, so the advantages of a "framework not tied to Flex" don't really work for me.) I really like the XML-based configuration that Mach-II, Spring/ColdSpring, and other frameworks provide. I know that some people hate coding in XML, and it does have its downsides, but I think that the simplicity and clarity it brings to the process outweigh the large XML files. As fortune would have it, there are two good options that have made their way from private alpha in to public beta: Mate and Swiz. Mate is from the team at ASFusion, who brought us some really cool Flash tips and tricks as Flash, Flash Remoting and Flex began to be integrated with ColdFusion. Swiz is the brainchild of the very smart Chris Scott, who brought us the excellent ColdSpring framework (a requirement for anyone building CF apps these days). In my initial explorations of these frameworks, I think the strength of each is as follows:

Mate uses an event map to describe the flow of events/actions within the application. I really like that and it makes a ton of sense, especially coming from an XML-based framework like Mach-II.
Swiz uses Inversion of Control and beautiful and handy things like annotations to significantly reduce coding and elegantly manage dependencies within your application.

Mate also offers all sorts of nicely encapsulated helpers for remoting and binding and dependency injection which make it pretty attractive. I don't know enough about Swiz at this point to say if it has all the same helpers and conveniences (and honestly, the documentation for Mate helps me understand a lot more about that framework that Swiz as Swiz currently lacks an equal amount of documentation). So given those conveniences and my comfort with the XML-based configuration/"event map," I think Mate is the way for me to go, for now. I'll blog about my experiences using Mate as I build apps with the framework. I'm personally very thankful that such smart people have provided the developer community at large with these great, powerful, time-saving tools.

Designing the Obvious Moment

Fri, 12 Sep 2008 09:37:00-0400

Robert Hoekman Jr. has written two quite excellent books on user interface and interaction design. I read these books earlier this year, but thought I'd finally get to blogging about them as part of my ongoing book reviews. Designing the Obvious was the first of the two books and, for me, the more successful of the two. It's rich with UI design aphorisms that simply make good common sense. Written in a simple, conversational style that draws heavily from actual Web application development experience, this book, and its counterpart just make sense. The book deserves to be read by anyone who actually builds any kind of Web application user interface, no matter how simple or complex. One of the things I really liked about his approach was the whole concept of turning beginning users into intermediate users quickly. That was the inspiration for my whole series of posts on the Contextual Guidance API. His outright demand that you make it simple, eliminate anything unnecessary in the app and in the design, and focusing on how you accomplish a task at every moment within the application is great advice. My team has taken that advice to heart and has begun designing our new applications around much of Hoekman's advice in this book. The result has been applications that visually make more sense to our clients, and clients that are really happy with what we're developing. Designing the Moment is focused on very common, very specific moments of interaction with a Web application and how they can be made better. It's less cohesive as the previous book, which took more of a big-picture view to the art and process of designing user interfaces. In this book, the examples are very specific, somewhat less generalizable, but many of the same principles from Designing the Obvious come in to play. My sincere feeling is that a number of the solutions presented in this book will become outdated over the next couple of years (ie; blog and social networking site design) but, again, the basic principles of clarity, task-orientation and removing all but the essential are more than sound. Designing the Obvious should be mandatory reading for all Web application developers, even those who rarely work on the UI. At the end of the day, the interface is the application, no matter what application you're using. Knowing how to create great interfaces is key to building great applications.

The Interesting and Not So Interesting of Google's Chrome

Tue, 02 Sep 2008 10:46:00-0400

I'll readily admit that explaining the idea behind the Google Chrome project via a comic book created by the genius Scott McCloud (whose book "Understanding Comics: The Invisible Art" should be required reading for anyone who does any kind of visually creative work) is great. Licensing it via Creative Commons is even better. But I'm not sure I get the need for another Web browser. If anything, I see this as merely a way for Google to continue to drive traffic to Google properties and bring a some "innovation" to the table in the process. If anything, it gives them an opportunity to start displaying "Works Best in Google Chrome" buttons on all their Web properties, just at the point that we've begun to collectively move away from Web applications that only work in IE or Firefox. I'm pleased that Google has chosen to use WebKit for the rendering engine, so if a page looks fine in Safari, it'll look fine in Chrome. However, they're using their own JavaScript engine, so Chrome becomes yet another Web browser on which you must test your applications. Looking at the features of Chrome that Google itself highlights in the overview comic book, some of their innovations just don't seem compelling enough to warrant another browser.

Stability: Safari, for me, is a very stable browser. Firefox and even Internet Explorer aren't half bad either. Browser crashes in this day and age are usually the result of a) out of control memory consumption by bad JavaScript, b) plug-in interaction gone haywire, or c) an error thrown at the operating system level. Chrome will address a) by having each tab be individually threaded and sandboxed, which will prevent one bad application in one tab from bringing down the browser — but that means applications can and will still crash, such as writing an email in GMail (the example used in the comic book). The plug-in interaction really can't be handled unless Chrome bans plug-ins, which is unlikely. Chrome itself is built on top of other operating systems (Windows, Mac, Linux), so it's still going to crash when there's an OS issue. No big win here, other than the individually threaded tabs.
A Faster JavaScript Engine: Um, SquirrelFish? Tamarin? SquirrelFish is going to be part of WebKit, and Safari 4.0, and it's mighty fast. Firefox 3.1 will contain a new, very fast JavaScript engine. Processors get faster and so JavaScript processing on those processors gets faster. Google's new JavaScript engine (V8) may be faster, and it brings to the table some interesting compile-time optimizations. It's going to generate machine code rather than run interpretatively, but I wonder if this will only work if you use Gears and the GWT. If you're not required to use Gears +/or the GWT, then Chrome still have to compile and translate the JavaScript in to machine code when the page is loaded, generating a slowdown on page load times. At the end of the day, to the end user, will it really be remarkably faster, especially as SquirrelFish and the other JavaScript engines are open-source projects with lots of smart people working on them from all over the world? If V8 turns out to be a truly excellent JS engine, then it's nice that it's going to be available for other browser makers to use. Time and performance studies will show if it's useful and adaptable for others to use.
Security: Sure, IE 6 is very insecure. IE 7 is better, and there are exploits available for Firefox and Safari. But those browsers are pretty dang secure compared to where we were a year or two years ago. Apple may not be the best when it comes to openness and fixing exploits on the same day they're discovered, but they're not terrible, and serious exploits are fairly uncommon. Phishing detectors are built in to Firefox and IE, though sadly not Safari, so that's nothing new. The sandboxing Chrome brings to the table is interesting, especially as plug-ins (a common source of browser crashes and Windows-based exploits) are going to exist outside and "above" the sandbox for each individual tab. I wonder, though, if plug-in makers (ie; Adobe) are going to have to rewrite their plug-ins so they work via this new kind of sandbox. Are there improvements here? Yes, in the sense of the double-sandboxing of plug-ins, and the simple sandboxing of each page so that keyloggers or other malware can't see what you're doing in a given tab. That's a nice win for consumers.
User Interface: Um, Safari's interface? Firefox 3's interface? How much simpler can you get with a default UI and still be able to click through basic tasks in a browser? The "nine most recent or related pages" view by default? No thanks, I'll go where I want to go.
Open Source: Well, WebKit, the core of Safari, is open source. The Mozilla project is open source. The final result may not be entirely open source, but how is this a big advantage or difference for Chrome?

I'll readily admit that the "Each tab gets its own thread/processing space" is rather nice. It'll make things work more smoothly. I'd imagine, however, that it wouldn't be too hard for this to be added to Safari, Firefox, or even IE. I also like the idea of the Task Manager for the browser, so you can see what resources are being consumed by each tab and kill tabs that are problematic. (Though I do find it interesting in their illustration of this in the comic book that they point to plug-ins (read: Flash) as the real problem with memory bloat and crashing Web browsers.) And Google wouldn't be able to make the Task Manager work inside of someone else's browser, so they needed to write their own to accomplish that task. I do appreciate the fact that Google is giving a wake-up call to browser makers. I do appreciate competition. I do appreciate the different approach they're bringing to the JavaScript engine and the double-sandboxing of tabs and plug-ins. I'm still not convinced that this is a way of driving additional traffic to Google properties (well, of course it is) and that we won't start seeing "You must use Google Chrome to access GMail. Download now!" appear in the not-too-distant future. I also hope that this doesn't hurt the development of Gears for Safari, or Firefox, as I think that's an important project and the tools it brings to the table should be available across Web browsers. I'm also not convinced that the average user will switch away from Internet Explorer. It's the default, it's what they're used to using, it acts in a way that they are used to, and users, quite simply, hate change. Google is a trusted brand, but that doesn't mean they're going to get my parents, or my co-workers, or the students at the University where I work to change from clicking on that "e" on their desktop to "launch the Internet." If they're going to take market share away from anyone, it's going to be Firefox and Safari, and, to a far lesser extent, Microsoft's dominant Web browser.

On Creativity and Creative Culture

Tue, 26 Aug 2008 08:53:00-0400

Ed Catmull, the cofounder and president of Pixar Animation Studios, wrote an excellent piece for the Harvard Business Review on developing a creative -- no, inspired and inspiring -- business culture. It's an excellent read and totally applicable to any number of industries, from movies or theater to software development. Catmull's central premise -- that you must hire truly creative people and give them free reign to succeed or fail wildly, and that everyone involved in the process has value and voice and must be allowed to contribute -- speaks directly to my years of running a theater company and the thousands of hours I put in to directing. One of the very early lessons I learned is that I never had all the answers. Yes, there are lots of directors as dictatorial auteurs out there, but some of the best moments and best ideas in the productions I directed didn't come from me. They came from the set designer, or the stage manager, or the actor in the bit part, or the crew member who saw something, something inspiring and creative and right and felt they could stand up and say something to me and I, in turn, was not so stupid as to pass up their great idea because it didn't come from me. I learned you had to foster that environment from day one, and if you did, you'd get a final product that was about 100% better than a product you tried to design, develop, and deliver with autocratic power. There are limits to this approach, of course. Someone has to take final accountability at the end of the day. Someone (or a group of someones, in Pixar's case) needs to be able to say "You know, that's interesting, but it's just not going to work." You still need leaders and teams running the show from the top, but if everyone who remotely touches the project feels that they have a voice, that their voice matters, that we are all equal and creative and working towards the same goal, you inspire passion, inspiration, and, sometimes, greatness. That's key to building a successful team, regardless of your industry. Pixar, of course, has the advantage of money, time, reputation, and some real geniuses running the show. But the same principle applies if you're running a 68-seat theater company or building complex enterprise software. Don't discount the suggestion of the administrative assistant who uses your software or the stage crew member who has to bring on and off 38 pieces of furniture each night. They're involved. They have ideas. Listen and make them feel valued, and you'll end up with something much better than you could have ever made alone.

SOA in Practice: The Art of Distributed System Design

Thu, 21 Aug 2008 14:54:00-0400

Thanks to the hour a day I spend riding mass transit to and from work (instead of the two hours a day I'd spend sitting in my car), I get to read a lot. I'd like to think that I'm spending most of this reading time on contemporary fiction, but that's really not the case. I'd estimate that a good 80% of my time is spent reading technology articles or books, or Entertainment Weekly. =) Call me crazy, but I do read a good number of technical books on the subway. I've not yet delved in to the arena of books reviews, but there's no good reason for that, especially with some of the excellent books I've read of late. "SOA in Practice: The Art of Distributed System Design," by Nicolai M. Josuttis, is a great introduction to the complexities of designing and deploying a service-oriented architecture for your business application needs. It's largely devoid of code, which some developers may find frustrating. I think that's a good thing, as SOA is about approach and not necessarily about the code it takes to set up and run those services. I suppose you could call this a book for managers, or system architects who are just looking to get in to service-oriented architecture design, but I think it provides such a thorough, clear introduction to the idea of SOA (free from the marketing jargon of companies trying to sell you their SOA "solutions") that it's valuable for any developer who wants to create more loosely coupled services inside a single application or across applications. After reading endless issues of eWeek and Information Week and CIO Insight, I certainly knew some of the principles behind the idea, some of the SOA bus solutions available, and how governance was really important to SOA rollouts, but as for the practicalities of how one might go about doing this in the real world? Not so much. Aside from the clear insight Josuttis brings to the book from his own SOA experiences, it's the lessons learned the hard way that I found to be most useful. Josuttis brings up a lot of issues with messaging and idempotence that I had simply not considered before and now form a cornerstone of how I'd look at developing a truly service-oriented architecture. That alone is worth the cost of admission. Perhaps its lack of code will lead some reviewers to say this book is more for pointy-headed managers than developers (even I wanted to see a bit more on how he handled the messaging patterns he raised with some code examples, or some kind of concrete solution rather than saying "This is out there, deal with it"), but I think it's a darn fine and thorough look at getting started with SOA.

News and Tidbits on ColdFusion 9 (aka Centaur)

Thu, 24 Jul 2008 13:00:00-0400

I just came across two blog posts which highlight some probably and some possible features for the next version of ColdFusion. My friend Adam Lehman (on Adobe's CF team) made an interesting post about the process that Adobe uses to develop new features for new versions of their products. It's a really interesting read, and the Synchronous Development process they use sounds quite interesting. You can't get the full details on that process from the SyncDev Web site because, well, their business is to sell consulting services. The key process idea of "Sell, Design, Build" (rather than "Design, Build, Sell") is a really interesting one as it ensures that your customers (and people you want as your customers) would actually want to buy your product before you ever build it. That's pretty powerful stuff for ensuring that your customer base will actually go out and buy your product. This is especially true of software, where versions above #4 tend to be about adding "nice" features when the product has already solved the core problems it was meant to solve or where the product has a free (often open-source) alternative. While Adam doesn't talk about anything that hasn't been announced elsewhere, it's an interesting and valuable read to understand not only Adobe's product development process, but where the ColdFusion team, specifically, is coming from in determining the product's future. Brian Rinaldi has put together a much more extensive look at all the public knowledge about ColdFusion 9. From process improvements to actual feature descriptions, his overview is a great read for anyone interested in the next version of the product and the language. I found it particularly interesting that it appears that Adobe has not gone the route of the Active Record pattern with their Hibernate integration, which I think may not save developers a ton of time but will instead allow for more flexible, robust uses of Hibernate from within ColdFusion 9. Hibernate is going to save developers a lot of time as it is, so I don't think there's a big loss in productivity by not going the Active Record route. (Implicit getters and setters will be a huge time saver, however.) The big unknowns are the "management features," as they're sometimes called. Those are the easy-to-grasp, sexy, obvious features which make it much easier to sell a product upgrade to managers. This could include a real ColdFusion IDE, or audio/video management tools, workflow and BPEL engine integration, or who knows. We'll know a lot more by the time MAX is over at the end of November. I'm looking forward to the time we'll finally get to play with CF9!

Doing the Upsert

Wed, 21 May 2008 11:24:00-0400

In the process of developing the contextual guidance API, I tried to push as much of the work as I could to the database. I knew that I would have to do multiple queries to check and see if a user had an existing record, when their last visit was to the application, and more. I had read some time ago about the concept of the upsert, and thought this was a good time to put it in to practice.

In many applications, you need to check to see if a record exists in a table, and do an UPDATE if a record does exist, or an INSERT if one does not exist. This is a common pattern in application development. Many developers do something like this:


Query the table to see if a record exists which matches the passed ID
IF match found {
	Run an UPDATE query
} else {
	Run an INSERT query
}

That's fine, but ultimately requires two trips to the database, or more, depending on what other conditional processing you need to do. The upsert combines all of this in to a single SQL statement. In addition, I needed to run some other database-related logic that would affect the SQL statement. If a user hadn't visited the application in more than 180 days (6 months), I wanted to treat them as a new user and delete their previous record in the log table. (You could set an "isDeleted" flag or delete the record, that's up to you.)

My solution is as follows, and combines some conditional logic in ColdFusion with conditional logic in the SQL as well. This is for MS SQL Server, but the concept applies to pretty much any database.



DECLARE @lastVisitDate datetime
set @lastVisitDate =
	(
	SELECT lastVisit
	FROM logTable
	WHERE userID = 
	AND appName = 
	
		AND sectionName = 
	
		AND sectionName IS NULL
	
	AND DateDiff(n, lastVisit, getDate()) > 30
	)
			

IF @lastVisitDate IS NOT NULL
	BEGIN
		
		IF DateDiff(d, @lastVisitDate, getDate()) > 180
		     BEGIN
				DELETE FROM logTable
				WHERE userID = 
				AND appName = 
				
					AND sectionName = 
				
					AND sectionName IS NULL
				
		     END
					

		UPDATE contextualGuidanceLog
		SET totalVisits = totalVisits + 1,
			lastVisit = ,
		        previousLastVisit = (
						SELECT lastVisit FROM contextualGuidanceLog
						WHERE userID = 
						AND appName = 
						
							AND sectionName = 
						
							AND sectionName IS NULL
						
						)
			WHERE userID = 
		        AND appName = 
			
				AND sectionName = 
			
				AND sectionName IS NULL
			
					
		
		IF @@rowcount = 0
			BEGIN
				INSERT INTO contextualGuidanceLog (
					userID,
					appName
					, sectionName
					)
					
					VALUES (
						,
						
						
								, 
						
					)
		      END
	END

This may look a bit complex, but the key thing here is the UPDATE followed by the IF @@rowcount = 0 statement. If no records were updated (meaning there was no record to match the passed values), then we do an insert. This is just like our original logic, except we're doing everything in one query, rather than two. Because I also needed to check the last time a record for this person was updated and make sure we're not updating it more than once every 30 minutes, I combined 3 potential <cfquery> calls in to a single one.

There's one issue, though, that I'm not too happy with:

I realize that I'm repeating a bunch of code to generate the WHERE clause in these SELECT statements. I can't generate this text dynamically as a string and then <cfoutput> the WHERE clause because I'm using <cfqueryparam>, as I damn well better be. I guess I could build a dynamic string and then wrap it in the Evaluate() function, but that seems pretty ugly to me (and may not work because of how CF's engine parses the query text and does the binding to the values in a <cfqueryparam> tag). Though I guess using Evaluate() isn't any uglier than the repeated code for the WHERE statement.

I like the upsert approach. I'll be using it from here on out in these situations. If anyone has any suggestions about the repeated WHERE clause code, I'd love to hear them!

Wrap Up on the Contextual Guidance API

Wed, 14 May 2008 15:49:00-0400

Given all the background thought I've done on the contextual guidance API, it was a breeze to implement. Planning before coding really does pay off!

The table I'm using to store the data for the tool looks like this:

As you can see, I've simplified things and haven't gone the route of more advanced, multivariate analysis on usage patterns for a users within an application. I certainly could have, but given that determining a user's experience level with an application is an act of generalization and not a precise science, this seemed right. Or maybe I'm lazy, or like things simple, or both.

In addition to a totalVisits value, I'm storing two date values per user, per application, per section of the application (if that's passed). This allows me to place some context on the user's activity within the application. Instead of just saying "oh, well, they haven't visited this application in 30 days, they should be given the beginner materials again," I can instead say "Well if they haven't visited in more than 30 days, but less than 90, and their total visits were greater than 50, give them the intermediate materials because they probably remember some of what's going on here." Again, in an ideal setup, I should look at not only the number of visits (or visits per section), but average visits over periods of time, the length of breaks between visits, and the exact actions they took within the application to provide them with highly specific contextual guidance. I think that for my users and my applications, being able to look at their activity within sections of a complex application given a rudimentary temporal context will be just fine. If anything, this simpler version of the logic will result in a more consistent display of help/guidance as the user moves from beginner to intermediate user to expert. When you try to get super-specific about guessing what a user is trying to do and provide help each step of the way, you are in danger of ending up with Clippy.

There will be one more related post about the API. It has to do with the magic of upserts and how I thought moving some business logic in to the query itself would save me code, but may not have.

The Logic of the Contextual Guidance API

Tue, 13 May 2008 13:59:00-0400

In my last post, I provided background on this project, and why it's useful to be able to provide users with help that fits their experience with the application. I'd like to shift now to looking at the business logic that would make this happen.

I've already covered the method signature for this very simple API, but how does the API know to return a value of "beginner," "intermediate," or "advanced,"? To make this determination, we have to think beyond a simple, reductive value. I could certainly write something simple to say that "If someone has logged in to the app more than 20 times, they've got to be an intermediate user." That's easy, but misleading.

People tend to forget things. I work in academia, and work here is highly cyclical. There's a flurry of activity at the start of each academic term, and the place is a ghost town in the months of May and June (well, faculty and students can't be found. Staff is working as hard as ever). Faculty and TAs working on courses tend to do a lot of work in a concentrated period of time, then walk away and don't look back until the next time they work on a course. I think that many users are the same way: they get what they need to get done ASAP, then walk away without a look back until the next time they need to use the app. There are exceptions, of course (email clients, social networking and financial sites come to mind). Most of the time, though, a user will need to use your application for a little bit and then come back to it only when needed again. In this away time, users forget a whole lot of stuff. Humans are creatures of habit and need reinforcement to retain processes. If someone uses your application for two weeks and then doesn't use it again for eight months, they're going to have forgotten how to do things. They won't be starting entirely from scratch, as, ideally, bits and pieces of how the application works will come back to them, but they'll definitely need a refresher.

The contextual guidance API needs to take simple forgetfulness into account. We can't just say "If someone has logged in to the app more than 20 times, they've got to be an intermediate user." We have to take into account time — specifically, the time between their last visit and their current one. But we can't stop there. What if they visited our app a lot nine months ago, but then came back to it for the first time yesterday. Are they still an intermediate user? Have they remembered all the basic stuff that they needed to remember? Probably not. So we need to look a bit at their history and factor that into our decision.

One possible version of the business logic would work as follows:

Store a record of a request for a page in the app
- Insert if it's a new user
- Update if it's an existing user
Look at two stored values: lastVisit and totalVisits
Run a simple decision tree based on these values (see below for discussion)
Return "beginner," "intermediate," or "advanced," as appropriate.

So what would the decision tree look like? There are a number of ways of approaching this, but I plan to say something like this:

If lastVisit < 14 days and totalVisits < 10 ===> new
If lastVisit < 14 days and totalVisits between 10 and 25 ===> intermediate
If lastVisit < 14 days and totalVisits > 50 ===> new
If lastVisit > 14 days and totalVisits < 50 ===> intermediate
If lastVisit > 30 days and totalVisits < 75 ===> intermediate
If lastVisit > 180 days ===> new

There are lots of choices you can make here in terms of the "rules" for determining someone's experience with an application. But we're only making decisions on two pieces of data. It's useful, as Edward Tufte might say, to be multivariate. We need to factor in previous visits to the application and not just look at the lastVisit value. What if someone's lastVisit was yesterday, their totalVisits was 64, but the previous visit before yesterday was 9 months ago? Should they really still be labeled an "intermediate" user even though they haven't been using the app for more than a single day in the past nine months?

Beyond examining simple date values, we could also look at the amount of time the user spent working on the app in the past few days. "The amount of time" is always a highly flawed piece of data on the Web, as a user can sit on a Web page for hours but not do anything during that time. Instead of tracking hours, minutes, and seconds, we can look at the number (or even kind) of requests a user is making in our application and factor that in. You could even go so far as to weight certain, difficult tasks more than others. I'm not going to go quite that far, but it's another option. Looking at all the requests a user made and not just their number of logins can help determine if they're really using the app, or just browsing.

What about in-house staff or "experts"? Should they be a part of this decision tree, or should they always receive the "advanced" view? That's probably outside of the scope of this simple API, as the application that's calling this API can decide if the user is an expert and always return "advanced," if appropriate.

So that's a look at the business logic of the contextual guidance API. Next up will be a look at the (simple) database table structure, and some sample code in ColdFusion, my development language of choice.