Simple Hot Deployed Fuse Routes

August 26th, 2010

If you have spent some time looking at or working with Fuse ESB then you will be aware of what powerful option its pluggable component architecture represents. If you have not come across Fuse ESB before then Fuse Source describe it as:

Based on Apache ServiceMix, FUSE ESB is an open source enterprise service bus (ESB) that provides a standardized methodology, server, and tools to deploy integration components, freeing architects from the dependencies that have traditionally locked enterprises into proprietary middleware stacks.

During the past year Answer have worked on a number of Fuse based implementations and have picked up several tips and tricks which are helpful to both the development and operations teams. Over the next few articles we will be capturing some of these handy hints and trying to highlight how they can be used.

The Camel routes that form the core of Fuse can be described using both the Camel DSL and Camel XML formats. If your defining a Camel route during the course of your project then the DSL is certainly my preference as it has a number of strong advantages over the XML format; including greater IDE support, easer route reuse, enhanced debug support and ease of reading. That said, thanks to the ServiceMix hot deploy capabilities there are a couple of situations where the XML routes can be extremely useful. It is this that we will look at in this article.

Camel XML Routes

The Camel XML language utilises Spring namespaces so that they can be defined within your Spring configuration files. The simple example below takes a file from msg-in directory, puts it on the MESSAGE.TRANSPORT queue and then saves it back to the msg-out directory under its original filename.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:c="http://camel.apache.org/schema/spring"
 xsi:schemaLocation="

http://www.springframework.org/schema/beans

http://www.springframework.org/schema/beans/spring-beans-2.5.xsd

http://camel.apache.org/schema/spring

http://camel.apache.org/schema/spring/camel-spring.xsd">

 <c:camelContext>
   <c:route>
     <c:from uri="file://tmp/msg-in/"/>
     <c:to uri="activemq:MESSAGE.TRANSPORT"/>
   </c:route>

   <c:route>
     <c:from uri="activemq:MESSSAGE.TRANSPORT"/>
     <c:to uri="file://tmp/msg-out?fileName=${id}"/>
   </c:route>
 </c:camelContext>

</beans>

What you might not be aware of is that if you drop this XML file into the fuse deploy directory then ServiceMix will hot deploy this route without the need to package it inside an OSGi bundle. Now this is certainly not the recommended approach for deploying your business critical routes, as properly bundled, released packages, that are managed using a features file is a safer way to create a controlled, reproducible infrastructure. However, if used properly then it can be a very powerful option.

So how can I use this?

Lets look at a couple of situations where this might be very useful in both test and production infrastructures. Consider a test deployment where one of the external message queues is not available and you want a simple stub service without having to deploy an externalised consumer (e.g. Performance testing). In this case you could hot deploy a XML based route to consume these messages such as the snippet below. Using a hot deployed message consumer in this case allows you to build a standard environment of released software components without the need for customisation of your core application code.


 <c:camelContext>
   <c:route>
     <c:from uri="activemq:MESSSAGE.TRANSPORT"/>
     <c:to uri="file://tmp/msg-out?fileName=${id}"/>
   </c:route>
 </c:camelContext>

Now lets consider another example in the production environment. Lets say that for what ever reason there has been a problem with some of the messages passing through the bus and as a result they have been placed onto the error or dead letter queues. These messages are important and they need to be rectified so that processing can continue. This could easily be achieved using the hot deployed XML route option. The snippet below could be provided to the Operations team as a means of taking the messages off the error queue and saving them to a directory.

 <c:camelContext>
   <c:route>
     <c:from uri="activemq:INVALID.MESSAGE.QUEUE"/>
     <c:to uri="file://tmp/fuse-esb-error-out?fileName=${id}"/>
   </c:route>
 </c:camelContext>

Once in the directory then they can be edited to resolve the issue before a second route puts them back onto a queue so that the processing can continue..

 <c:camelContext>
   <c:route>
     <c:from uri="file://tmp/fuse-esb-error-resolved"/>
     <c:to uri="activemq:MESSAGE.QUEUE"/>
   </c:route>
 </c:camelContext>

Once deployed these temporary routes can be stopped/started/undeployed as any other bundle using the ServiceMix console.

Summary

Hopefully these simple examples have demonstrated the potential power of the XML routes and ServiceMix hot deploy capabilities. That said, this is certainly one of these capabilities that falls into the ‘just because you can, doesn’t mean you should’ set as it has a lot of great potential but if misused then it could also lead to an over complex, uncontrolled environment. Consider the pro’s and con’s before taking this approach and if it still seems the right thing to do then go with it.

ServiceMix bridge to WebSphere MQ

August 24th, 2010

A couple of the customers we’re working with on Fuse (ServiceMix) implementations today have a need to integrate with, or bridge from WebSphereMQ systems. In this post I’ll explain how to create the connection to WMQ and how to set up a simple bridge from ActiveMQ using Camel.  It covers a couple of the gotchas that we found along the way.

Our user story is:  “We want messages on the WMQ.OUT (ActiveMQ) queue to appear on the AMQ.IN queue in WebSphereMQ”.

You can do this type of bridging natively using ActiveMQ, but for the project this came from, we wanted a runtime/re-deployable Camel route for the Fuse container.  If you prefer the native bridging route, some of this article will still apply.

The Camel routing part of the requirement is straightforward, it will just be a line of DSL or equivalent XML (as shown) that takes the form;

 <camel:camelContext id="camel">
 <camel:route>
     <camel:from uri="activemq:WMQ.OUT"/>
     <camel:to uri="wasmq:AMQ.IN"/>
 </camel:route>
</camel:camelContext> 

The tricky bit is setting up the “wasmq” component to work correctly as a connection to the WebSphere MQ server. First, you will need to obtain the WASMQ JMS client libraries. This is proprietary IBM code and can only be redistributed under an appropriate license from IBM. Seems awfully silly to me to make it difficult for people to access your expensive server infrastructure, but there you go. If you have the appropriate permissions, the three files you will be interested in are;

  • com.ibm.mq.jar
  • com.ibm.mqjms.jar
  • dhbcore.jar

You will probably find these buried in the enormous tar or zip file that constitutes the IBM JMS Client library download.  For this exercise, you need nothing else from that download, and in particular, you need to install nothing or make any modification to your machine for this to work.

Despite all the IBM bluster about how they are the world’s foremost OSGi company, very few of the JAR files they ship seem to be OSGi bundles, and these are no exception at the time of writing.  To use these files in the Fuse/SMX 4.x container, you will need to wrap them.  You can do this at the Fuse command line like so;

osgi:install -s wrap:file:/path/to/com.ibm.mq.jar

.. repeating for the other two. If you’re familiar with creating your own SMX features, you can add these to a features file and manage it that way which is probably a good idea.  I’ll do a later post on features.

With those deployed, you should see similar to the following in the console depending on where your files on disk were located;[

karaf@root> osgi:list
...
[93] [Active ] [ ] [ ] [60] wrap_file_jms-client_com.ibm.mq.jar (0)
[94] [Active ] [ ] [ ] [60] wrap_file_jms-client_com.ibm.mqjms.jar (0)
[95] [Active ] [ ] [ ] [60] wrap_file_jms-client_dhbcore.jar (0)…

If that's the case, then so far so good.  We can start to configure a connection with the WAS MQ server.

<bean id="wasmq" class="org.apache.camel.component.jms.JmsComponent">
 <property name="connectionFactory">
  <bean  class="com.ibm.mq.jms.MQQueueConnectionFactory">
   <property name="transportType">
    <util:constant           static-field="com.ibm.mq.jms.JMSC.MQJMS_TP_CLIENT_MQ_TCPIP" />
   </property>
   <property name="hostName" value="${wasmq_hostname}" />
   <property name="port" value="${wasmq_port}" />
   <property name="queueManager" value="${wasmq_queue_manager}" />
   <property name="channel" value="${wasmq_channel}" />
   <property name="useConnectionPooling" value="true" />
  </bean>
 </property>
</bean>

The values for hostName, port, queueManager and channel will be based on the configuration of the AMQ.IN queue in the WAS MQ server.  They can be read into the XML using a Spring PropertyPlaceholderConfigurer or typed in directly.

This completes the basic part of the connection configuration for the WAS server, and the XML file (see later) can be dropped into the Fuse ./deploy directory to set the bridge up.  But we found a fairly serious problem when we started using this.  The customer (whose WAS MQ server we were pushing messages to) reported that Fuse was opening and closing a connection for every message sent.  This is very inefficient and particularly bad for us as we are expecting around 200 messages per second to go across this bridge.

Because Fuse/SMX's JmsComponent uses Spring's JmsTemplate under the hood, herein lies the issue.  JmsTemplate expects that your ConnectionFactory will use connection pooling, and therefore will create a new connection/session for each call to one of its send() methods.  The IBM documentation appears to indicate that connection pooling is enabled by default in their JMS client, so what's going wrong?

Well, we can't legally view the IBM code in the JMS client, but through trial and error testing and the available IBM docs, we discovered that calling a static method in the MQ code correctly initiated connection pooling such that JmsTemplate would re-use pre-created connections and for our purposes only ever open one connection to the WAS MQ server.  The method to call is;

com.ibm.mq.MQEnvironment.addConnectionPoolToken();

and the following short program demonstrates the issue.  It has dependencies on the 3 IBM JMS libs discussed earlier, plus jms.jar from the JEE SDK and/or a JEE client implementation library containing the javax.resource package.. I just used jboss-javaee.jar which has both..

package wasmq;

import javax.jms.Session;
import javax.jms.TextMessage;

import com.ibm.mq.MQEnvironment;
import com.ibm.mq.jms.*;

public class WasTest {

 public static void main(String[] args) throws Exception {

   MQQueueConnectionFactory factory =
    new MQQueueConnectionFactory();
  factory.setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);
  factory.setHostName("10.0.0.1");
  factory.setPort(1394);
  factory.setQueueManager("QUEUE.MGR.NAME");
  factory.setChannel("CHANNEL.NAME");
  MQQueue queue = new MQQueue("QUEUE.NAME");

   // connection pooling
  MQEnvironment.addConnectionPoolToken();

  System.out.println("Sending...");
  for (int i = 0; i < 5; i++) {
   MQQueueConnection  con = (MQQueueConnection)
    factory.createConnection();
  MQQueueSession sess = (MQQueueSession)
    con.createSession(true, Session.AUTO_ACKNOWLEDGE);
  MQQueueSender sender = (MQQueueSender)
    sess.createSender(queue);
  con.start();
  TextMessage msg = sess.createTextMessage();
  msg.setText("Hello #" + i);
  sender.send(msg);
  sess.commit();
  sess.close();
  con.close();
  }
  System.out.println("Done.");
 }
}

You can see from the code above that a new Connection is obtained from the factory on each iteration of the for loop. Without connection pooling, this will result in a connection per message. With the connection pooling setup, it won't and will only result in a single connection being opened against the server. So far so good - how do we call a static method in an arbitrary bundle from an XML file?  Spring to the rescue with a little known construct that has been around since the early days for exactly this sort of scenario..

<bean class="org.springframework.beans.factory.config.MethodInvokingFactoryBean"
 p:staticMethod="com.ibm.mq.MQEnvironment.addConnectionPoolToken"/>

When the Spring context is created, the static method will be called and connection pooling will start working for us! Finally then, here's the complete file that we use to create the bridge and which can be dropped into the deploy directory of your Fuse/SMX instance..

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:context="http://www.springframework.org/schema/context"
 xmlns:util="http://www.springframework.org/schema/util"
 xmlns:camel="http://camel.apache.org/schema/spring"
 xmlns:p="http://www.springframework.org/schema/p"
 xsi:schemaLocation="

http://camel.apache.org/schema/spring

http://camel.apache.org/schema/spring/camel-spring.xsd

http://www.springframework.org/schema/beans

http://www.springframework.org/schema/beans/spring-beans.xsd

http://www.springframework.org/schema/util

http://www.springframework.org/schema/util/spring-util-2.5.xsd

http://www.springframework.org/schema/context

http://www.springframework.org/schema/context/spring-context-2.5.xsd">

 <context:property-placeholder location="file:etc/was-bridge.properties"/>

 <camel:camelContext id="camel">
  <camel:route>
   <camel:from uri="activemq:WMQ.OUT"/>
   <camel:to uri="wasmq:AMQ.IN"/>
  </camel:route>
 </camel:camelContext>

 <bean class="org.springframework.beans.factory.config.MethodInvokingFactoryBean"
   p:staticMethod="com.ibm.mq.MQEnvironment.addConnectionPoolToken"/>

 <bean id="wasmq" class="org.apache.camel.component.jms.JmsComponent">
  <property name="connectionFactory">
   <bean  class="com.ibm.mq.jms.MQQueueConnectionFactory">
    <property name="transportType">
    <util:constant
     static-field="com.ibm.mq.jms.JMSC.MQJMS_TP_CLIENT_MQ_TCPIP" />
     </property>
     <property name="hostName" value="${wasmq_hostname}" />
     <property name="port" value="${wasmq_port}" />
     <property name="queueManager" value="${wasmq_queue_manager}" />
     <property name="channel" value="${wasmq_channel}" />
     <property name="useConnectionPooling" value="true" />
   </bean>
  </property>
 </bean>
</beans>

Once deployed, we can route messages in our Fuse container to the WMQ.OUT queue safe in the knowledge that they will be received on our customer's WAS MQ infrastructure on the AMQ.IN queue.  If we undeploy the bridge, or don't deploy it at all such as in our test environments, then the messages will remain on the WMQ.OUT queue in Fuse.  We may choose to deploy a temporary consumer, or test consumer to keep the queue clear during testing.

Happy bridging :)

Definition of Mobile Event – 10th September 2010

August 13th, 2010

Definition of Mobile

The Most Business Critical Event This Year.

We are delighted to be hosting the Definition of Mobile event this year exploring how mobile technology is shaping the way businesses do business.

With “Mobile Innovator” Steve Dobson leading the event we are looking forward to a host of successful organisations in attendance to see how the future of mobile is looking.

Who is Steve Dobson? – Over the last 20 years Steve has been involved in the design of systems and software for applications including medical, military and commercial and has seen plenty of technology ‘development’. Bringing his experience in media, mobile and internet technology gained with the likes of ITV, Sky, Macromedia, Pixar and Apple, he leads a world beating team of developers who create solutions and applications that transform the way the world does business.

“The world has changed. Mobile has brought a new market available to millions, this scalable platform encompasses application technology from Apple to Android, bringing with it a new customer that wants your product in their pocket. It has changed business, with new solutions and efficiencies through feature-rich innovation, enabling you to do more.”Definition of Mobile

The agenda for the event looks to focus on both consumer facing applications and enterprise/business apps. Steve claims “Mobile business applications have the potential to vastly improve productivity and performance…there is a place for using mobile devices across all companies from logistics to sales and we’ll be looking to explore these channels across a host of different organisations”

If you received an invitation to the event, then we look forward to seeing you. To RSVP  email michael.jessop (at) answerconsulting.com confirming your attendance.


Achieving Flow

August 10th, 2010

This is a file from the Wikimedia Commons.To achieve flow is to be happy. Some developers call it ‘the zone‘, others ‘hack mode,’ but all agree that it is the best place to be when writing code. It isn’t unique to Software Development. Getting things Done author David Allen borrows a simile from karate to describe it: “Mind Like Water“.

In his classic book ‘Flow – The psychology of Optimal ExperienceMihály Csíkszentmihályi describes the state of flow in great detail.

It is what the sailor holding a tight course feels when the wind whips through her hair, when the boat lunges through the waves like a colt – sails, hull, wind, and sea humming a harmony that vibrates in the sailor’s veins. It is what a painter feels when the colors on the canvas begin to set up a magnetic tension with each other., and a new thing, a living form, takes shape in front of the astonished creator.”

As a programmer I can relate to the painter’s experience. When I spend the day in my text editor writing code: moving from green bar to green bar, filling files with text and re-factoring complex structures into something clean and simple. The time flows by and I’m shocked to discover when I’ve run out of day.

Flow can be achieved by teams working together as well as individuals.  Surgeon Atul Gwande recounts an experience of a team achieving flow as they raced to save a patients life.  The clock was ticking and the patient’s chances of survival were reduced with every passing the second.  The assembled team had never worked together before and the patient’s life depended upon them coming together as a team.

They were swift, methodical, and in sync.  The case was far from easy, but for whatever reason, it seemed like nothing could thwart us….  Steve, thinking ahead, asked Jay to grab a retractor we needed.  Joaquim nudged me to make the abdominal incision bigger, and stayed with me at every step….  Because we worked as a single unit, not as separate technicians, the man survived….  The operation had been symphonic, a thing of orchestral beauty….  From the moment we six were all dropped together into this particular case, things clicked.  It had been almost criminally enjoyable.

The Checklist Manifesto

It may be possible for small teams of highly trained medical professionals to achieve flow in the well resourced operating theatre, but what about more realistic jobs. Can the factory worker achieve flow?   A recent episode of This American Life shows that they can.

The experience of General Motors and Toyota at the NUMMI plant in Freemont, California suggests that they can.  Episode 403 of This American Life tells the story.  General Motor’s  workers enjoyed a level of pay and benefits they could not achieve working anywhere else.    The job for life it provided was a gilded cage because the work itself gave no satisfaction.

The way the plant was organised made it hard for them to achieve flow, so they sought happiness elsewhere, even in drinking, gambling, prostitution and drugs.  Management just wanted them to do there job, and yet they could fail to do even that. Absenteeism was high and one worker deliberately sabotaged the cars.  With the power of the union they thought there jobs were safe no matter how badly they behaved.  They were wrong.  The plant was closed in 1982.

In 1984 the plant was reopened as part of a joint venture between General Motors and Toyota.  Most of the same workforce were sent to Japan to learn the Toyota Production System.  The Toyota Production System introduces flow to the production line based on it’s two fundamental principles: Continuous Improvement and Respect for People.  Following these principles allows not only supplies and materials to flow, it also allows the people on the production line to achieve a state of flow where they are challenged but not overburdened.

The workers were proud of the Chevy Nova they were building. One worker would leave postcards with his name and address under the wiper of Nova’s he saw around, asking the owners to drop him a line. Another worker would go to the Chevy dealership to stare at the Nova’s just sitting there on the lot.

It may be nice that the employees were happy in there work, but surely car factories exist to generate profit by making cars, not keep the workforce happy.  What were the benefits for General Motors?  Plenty: within three months the same plant with the same workers achieved near perfect quality at a much reduced cost. It was estimated that it would have taken 50% more workers under the old system to produce the same car.

Nummi may have closed and Toyota may be experiencing problems right now, but in Nummi they were able to prove that a large, industrialised workforce can achieve flow.  General Motors also discovered the huge commercial benefits of allowing the workforce to achieve flow.

NUMMI plant in Fremont with Mission Peak behind it. (Joint venture between General Motors and Toyota.

I work in Software Development and I wan’t to achieve flow. I want the end users of the applications that I help to create to achieve flow.  I want the business analysts and the testers and the help desk and the project managers to all achieve flow. In ten years time I want the maintenance programmers extending the code base to be achieving flow as they implement their change requests.

Is this an unrealistic goal?   I hope not.   Is there some amazing secret that has to be uncovered before it is possible? No, because none of these principles are new. They really just seem to be common sense once they are being followed.  The unfathomable thing is why everybody isn’t doing this already.  The second half of the This American Life episode provies a clue when it talks about General Motor’s failure to replicate what had happened at Nummi.   They would take pictures of every inch of the Nummi plant, and then ensure that another plant looked exactly the same. In every aspect they did things exactly the same way, and yet they failed to replicate the success. Why didn’t it work?

Csikszentmihalyi  explains the problem:

It would be erroneous to expect that if all jobs were constructed like games, everyone would enjoy them. Even the most favorable external conditions do not guarantee that a person will be in flow. Because optimal experience depends on a subjective evaluation of what the possibilities for action are, and of one’s own capabilities, it happens quite often than an individual will be discontented even with a potentially great job.

If Flow comes from within and the external conditions needed to achieve it are unique to each individual then copying the external environment will not copy the flow state. In each place the worker must be able to find what they need to achieve flow.  However, the Toyota Production System and the experience of Atul Gawande seem to contradict this idea. At Toyota work is standardised, with every activity scripted to the finest point of detail. Gawande found the use of a standard checklist helped even the finest surgical teams to find flow and avoid mistakes.

In this blog I will make my own personal attempt to understand these contradictions, evaluating the conflicting claims and attempting to untangle the apparent paradoxes. My goal is to produce the best software I possibly can, and I believe learning how to achieve flow in software development is the best way to achieve that goal.

Simple and Secure Artifactory Configuration

August 9th, 2010

If you are using Maven or a similar tool for your builds then a repository manager is a very useful addition to your build infrastructure.  The repository manager acts both as a proxy between the build tool and the outside world and as an internal repository for build artefacts.

When setting up a repository manager it is important to ensure that only authorised users are able to deploy released artefacts into the repository while ensuring that developers have access to snapshot artefacts when required.  This prevents unauthorised users from overwriting officially released artefacts which may then be pushed out into a production environment.

Below is a configuration process for Artifactory which I have used on several occasions to secure access to the repository.  This process provides proxied access to any remote repositories required by the development team, allows developers to deploy snapshot artefacts to the repository but only allows authorised users to deploy released artefacts.  The instructions are for Artifactory but the principles remain valid for other repository managers such as Nexus

Add Remote Repositories

Artifactory can proxy any remote Maven repository and will cache any artefacts downloaded from those repositories.  This improves build time for future builds since the artefact is now stored on the local network.  The majority of common remote repositories are configured by default in Artifactory but if your project requires any that aren’t (such as the Spring Enterprise Bundle Repository) they can be added.

  1. Login as an admin user
  2. Click on the Admin tab
  3. Click on the Repositories link in the Configuration menu
  4. Click on New in the Remote Repositories panel
  5. Fill in the details of the new remote repository
  6. Now add the new remote repository to the remote-repos virtual repository. Double-click on remote-repos in the Virtual Repositories panel
  7. Find the new repository in the Available Repositories list and click on the arrow to add it to the Selected Repositories list
  8. Click Save

Create User Groups

Create a new Group for users who are authorised to deploy to the release repository.

  1. Click on the Groups link in the Security menu
  2. Click New
  3. Enter the Group Name (e.g. deployers)
  4. Click Create

Setup Permissions

Create a new Permission Target to allow write access to the local release repositories for the deployers group

  1. Click on the Permissions link in the Security menu
  2. Click New on the Permissions Management panel
  3. Enter the target name (e.g. Deploy)
  4. On the Repositories tab, check the Any Local Repository box and uncheck the Any Remote Repository box
  5. Click on the left-pointing double arrow to remove the deselected repositories
  6. On the Groups tab, enable the Delete, Deploy, Annotate and Read permissions for the deployers principal
  7. Click Create

Create a new Permission Target to allow write access to only the snapshot repositories for anonymous users

  1. Click New on the Permissions Management panel
  2. Enter the target name (e.g. Snapshot Deploy)
  3. On the Repositories tab, uncheck both the Any Local Repository box and the Any Remote Repository box
  4. Click on the left-pointing double arrow to remove the deselected repositories
  5. In the Available Repositories list, select the ext-snapshots-local, libs-snapshots-local and plugins-snapshots-local repositories and click the right-pointing arrow to add them to the Selected Repositories list
  6. On the Users tab, enable the Delete, Deploy, Annotate and Read permissions for the anonymous principal
  7. Click Create

Create  Users

Create the users who will be allowed to deploy to the releases repositories add them to the deployers group.

The above configuration will allow anonymous users to deploy snapshots to the repository.  This is useful as it means that developers can deploy snapshots without requiring authentication configuration in their local Maven settings and minimises the user management on Artifactory.  However, if it is necessary to limit snapshot deployments to authenticated users too then a new Group can be added and the authorisation of the Snapshot Deploy Permission Target can be changed from anonymous users to the new Group.

Database Servers Are More Than Object Repositories

August 6th, 2010

Data storage is nothing new to a developer, we have numerous database platforms and endless frameworks for storing our business objects to them. One thing that does get forgotten however is the power that databases offer for manipulating our data over and above this object persistance.

Many developers having grown up with Nhibernate and other object storage framweworks may view database as a black box. This is an advantage in many circumstances, but creates a vacuum of knowledge regarding the functionality a good database can offer.

Recently we were asked to investigate  a companies batch wage report which kept timing out when processing. Upon investigation the module processing the wages report was performing over 100,000 single SQL statements. With some investigation we were able to summarise the report into a single SQL which ran almost instantly.

Advanced SQL can be a steep learning curve, but using it in enterprise .Net development can save you substantial development time and deliver a better performing solution.

5 Reasons To Choose Hosted CI

August 5th, 2010

Benefits of Hosted CI

1 – Access

Cloud hosted services allow world wide access, wherever you are. multiple development teams both geographically dispersed or time-shifted can access your build wherever and whenever. With Hosted continuous integration, development is not restricted to small teams in one room, you can have remote workstations submitting into repositories which can be accessed in the cloud, Globally.

2 – Security

Amazon’s EC2 secure cloud ensures that data is secure and backed up minimising  data redundancy from hardware or system failure. Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers as well as Amazon Data Centres providing secure storage like no other. Hosted services also offer operational security and network security to match ensuring that “your code” remains “your code”.

3 – Seamless Updates

SAAS providers have to ensure that their customers have a great experience with their product and as a result professional providers regularly update their services to make sure their technology progresses with the world around them. Hosted CI is no different, compatibility and integration is an evolutionary element of software development one which ensures the system works with current technologies.

With a hosted service you don’t need to manually install updates, worry about latest releases and versions, this is all handled by the SAAS provider allowing you to focus on your build, not your build environment.

4 – Removes Infrastructure Costs

A hosted service means you require little to no infrastructure, no machines to buy, install and maintain and equally no machines to repair and replace when they breakdown. There is no need for network configuration either, simply an internet connection and you can start your build in a matter of clicks.

5 – Hosted Support and Updates

Hosted services have forums and blogs keeping you up to date with the latest news about the service. effective forums and Q&A sections are almost an industry standard in the SAAS space. A professional service provider gives you the opportunity to submit questions and get answers in a timely manner ensuring you have very few issues and get the most out of your subscription.

MikeCI

Mike CIMike is a securely hosted, continuous integration web-based service for software builds that brings agile benefits to your  development teams, it shifts scarce development skills away from managing your infrastructure and back to your project, driving down costs and enhancing productivity, the service is designed for experienced builders and novices alike, with bootstrapping features and expert backup that minimise your learning investment. delivering rapid results and measurably increased productivity. Mike supports a range of industry standard build management tools for Java developers that enhance your operational flexibility and further reduces learning investment.

10 Things You Need to Know About Knowledge Transfer – Facts, Fiction, Myths Revealed!

July 15th, 2010

How do you define Knowledge Transfer and how will we know if we have achieved this?

Knowledge Transfer is the movement and application of knowledge. It is the sharing and communication of knowledge from a source, usually a Subject Matter Expert(s) within an organisation so that it is learned and applied by Subject Matter Recipient(s). Knowledge is taken to be transferred when learning takes place and when the recipient(s) understand the intricacies and implications associated with that knowledge so that he or she can apply it in context.

Knowledge is very complex

Knowledge comes in many forms and types. The most common distinction is between explicit and implicit – open referred to as tacit knowledge. Explicit ‘how to’ knowledge is easily shared between people and is re-enforced in valuable documented knowledge which can be used or applied. Implicit knowledge is experience based knowledge built up over time – the ‘why’ rather than ‘how’ knowledge. When asked, most people say that over 70% of their organisation’s vital knowledge is implicit. Yet knowledge sharing programmes focus on explicit knowledge or information. Why? Because that is the easier option if you do not have a structured approach and experience in drawing out this tacit knowledge.

Where does Commercial Knowledge fit in?

Commercial Knowledge may be explicit or implicit, or both, and is focussed on effective performance. Not ‘what is right’ but ‘what works’ or ‘what works best’ for a particular organisation. Key knowledge which can be shared and cascaded.

Knowledge Transfer is not new.

Knowledge sharing and transfer has existed since humans started to teach each other how to use a tool or plant a field. However, what is new is the structured approach required to achieve success in knowledge initiatives and to be able measure that success in terms of ROI.

Is Knowledge Transfer a fad?

Surveys which in 1996-97 showed the majority of managers citing knowledge initiatives as a ‘fad’, now show less than 5% believe so. Reasons such as the value and recognition of people resources as knowledge assets, the proven benefits of knowledge management and the momentum of the knowledge movement show that, however it might be labelled – Knowledge Sharing/Cascading/Transfer – its core strategies and practices are fundamental to the growth and flexibility of individuals and teams within organisations.

Knowledge repositories go beyond being merely libraries.

They act as knowledge connectors and aggregators, linking people to information and people to people. They manage content on intranets, help nurture communities of practice, play a key role in information resources management, and embed a culture of responsibility to ensure information currency.

People won’t share knowledge

Lack of a knowledge sharing culture within an organisation is often cited as a primary obstacle. However, in depth studies have shown this to be a myth. The real culprit is lack of time. This is the primary cause why knowledge does not get more widely shared. Time is not built into tasks and people’s roles to allow them to do this. There is also the requirement to support this process through mentoring and facilitation to ensure that this type of initiative does not become a ‘nice to have’, and therefore the first thing dropped because of work pressures. Knowledge Transfer programmes need to be facilitated and driven to ensure success.

Osmosis is a key factor in Knowledge Transfer

There is a mistaken belief that sitting in close proximity to a colleague over time will ensure that you will know what that person knows; as if some form of invisible knowledge link will form between two people. The fact is that you could observe or shadow a colleague for any period of time and still not know what they know. You could of course replay situations which you had observed, but you will not know why that person made the decision they made because that is based on their tacit knowledge. Our approach and expertise allow us to scope that knowledge.

There’s no obvious pay-back

* Company

* Subject Matter Expert

* Subject Matter Recipient

Those organisations who have embraced the concept of knowledge transfer have cited numerous cases where there are positive and proven benefits. These play out on three levels from a:perspective and result in a win-win situation for all participants. Contact Us and we can share some examples.

If you want to successfully transfer knowledge then you need a structured process and approach to include:

1. Identifying the knowledge holders within the organisation

2. Motivating them to share

3. Designing a sharing mechanism to facilitate the transfer

4. Executing the transfer plan

5. Measuring to ensure the transfer is successful

6. Applying the knowledge transferred

In today’s knowledge-driven economy, successful businesses sustain competitive advantage through the enhancement of knowledge, skills and the stimulation of innovation. Mindswap helps businesses to improve their competitiveness, innovation and productivity through the better use of knowledge and skills that reside within their organisation. Mindswap is a unique knowledge transfer approach which removes elements of chance and dispels any academic ideal to simply focus upon the knowledge that drives business benefit.

Mindswap is an award winning approach which was founded in IT and follows iterative and time-bound practises which delivers both efficiency and measurable business value. Adopting an active learning approach, Mindswap successfully builds precise technical, system and business knowledge without adding workload to your in-house experts. Our methodology has been recognised by quality bodies for the way in which it addresses business risk and competitive agility through providing a solution to key person dependencies and performance bottlenecks. We have also seen major transformations in team agility and responsiveness. Introducing and embedding the practice of Knowledge Transfer can support a cultural shift that improves an organisation’s ability to meet today’s rapidly changing business demands and create a “can do” culture.

A Calculus for Software

July 7th, 2010

Uncle Bob is looking for the Software Calculus, the new level of abstraction that will change the way we think about software.

Uncle Bob introduces the need for Calculus with Zeno’s Paradox regarding Achilles and the tortoise:

While it was intuitively clear that Achilles would pass the Tortoise quickly, the algebra and logic of the day seemed to suggest that the Tortoise would win every race given a head start. Every time Achilles got to where the tortoise was, the tortoise would have moved on.

The paradox highlights a limitation in algebra, and inability to deal with infinity.  Calculus provided a way of dealing with infinity that overcame this limitation and provided a means for solving a whole new set of problems.

There is an equivalent paradox in software development, and the calculus for software development must be able to resolve it.

The best code you will ever write is… code you never write.

He is right, we cannot keep piling complexity upon complexity, and sometimes that means more than simply writing better code, it means writing no code at all.  This means that we cannot look to programming languages or architectures for the next step because they exist to serve the purpose of writing code or building systems.  Sometime we want to do neither, sometimes it is better to do nothing.  Another discovery in mathematics even more fundamental than calculus was the discovery of zero, a way to represent nothing.  In software development we need a method that enables us to produce nothing.

This means that Model Driven Architecture, 4GL Database Languages and Quantum Computing cannot be the answer.  These are all advances that seek to provide a new way to write code, or have it generated or executed.  They are still, fundamentally, about producing code.  Calculus did not provide  a new way of doing algebra, if didn’t even make possible things that had been impossible using algebra.  While Issac Newton used his own idiosyncratic form of calculus to solve problems, he would translate the resulting ideas into a geometric form that his readers would more readily accept.  It provided a new way of thinking about problems.

The monster to by slain is complexity, but there is no silver bullet left that will allow us to put the beast to rest.  There are two types of complexity: essential and accidental.  Essential complexity is unavoidable, as Brooks explains:

The complexity of software is an essential property, not an accidental one. Hence, descriptions of a software entity that abstract away its complexity often abstract away its essence. For three centuries, mathematics and the physical sciences made great strides by constructing simplified models of complex phenomena, deriving properties from the models, and verifying those properties by experiment. This paradigm worked because the complexities ignored in the models were not the essential properties of the phenomena. It does not work when the complexities are the essence.

If the complexity of software is all essential, then is there no hope for improvement?  Back in 1987, when the essay was written, Brooks concluded by looking forward to what was to become Agile:

Therefore, one of the most promising of the current technological efforts, and one that attacks the essence, not the accidents, of the software problem, is the development of approaches and tools for rapid prototyping of systems as prototyping is part of the iterative specification of requirements.

It was the development of rapid prototyping and iterative development, in RAD and DSDM, that laid the foundations for the Agile movement.  Principles like Test Driven Development and YAGNI help us to avoid writing code that did not need to be written.

The question now is what next?  What is the next method for attacking the essential software problem while ensuring that the accidental problems do not return?  Personally I look to the principle of Lean for inspiration, not for new ways to write code but for new ways to see the problems that the code is attempting to solve.

Test driving the iPad

June 24th, 2010

We’ve had a pre-release iPad for a while now but our App Dev team have kept very tight hold of it.  The iPad really appeals to me as a home laptop alternative, and last week  I was fianlly able to take it home for the weekend for a test-drive to see what the family thought.

My wife’s initial thought was, “Oh dear, here we go again, Chris wants a new toy”! Am I really that transparent? Well, yes,  I am but…

Back to the review,

First impression is  “oh, is that it?”.If you have an iPhone it is all instantly familiar and even if you don’t it is very easy to use. My son had his email and instant messaging set up in 5 minutes and he is only 8! But impressions improve once you delve a bit deeper. The screen is lovely as is the fit and finish. It is a bit heavier than I was expecting – you really need to lean it on something when you use it. It is a perfect device for browsing the web from the sofa, answering the odd email and of course playing games. Mail, Maps and Safari give a really rich and satisfying user experience. Games are great, as are photos and movies. You can plug a keyboard in, you can get a usb adaptor, but like when buying a car all these are “optional extras”, and being an Apple device not exacly cheap!

Is it a “game changer” in the way the iPhone was? It is early days and Apple must be really happy with the initial sales and I guess we’ll have a better idea when other similar devices start appearing with a Google or Windows operating system. One thing is clear and that is the larger screen really gives developers a huge amount of freedom. I’m sure we will see apps appear on the device that change the way people think about how you can interact with a computer, I think the potential is huge. Our developers are certainly excited about the possibilities. Whether that will be restricted to consumer applications, or we start to see enterprises taking up the device will be very interesting. It is the sort of device where you can imagine a remote sales force using for financial planning meetings with customers at the home. Flicking through snazzy pension forecast graphs etc. Mind you I’d hate to be the guy trying to persuade the FD to purchase 400 iPads for the Sales team!

Which brings me to the price, it is expensive, and as usual in the UK it is considerable more expensive than anywhere else. People need to carefully consider if they need a 3G connection (probably not not if they are only likely to use the wifi at home), or can plug in an external hard drive and do with less storage.

Personally, I really liked it, but I’d still keep a laptop. My wife also, grudgingly, liked it but is tied to a real keyboard so I’ve got a bit more persuading to do before we can get one!

The kids loved it at first, but then found it very frustrating when trying to play games on their favourite websites. Lack of Flash support is a major annoyance. If Flash can run on a Mac why can’t it run on an iPad or iPhone? I think we all know the answer but in the short term it is the users that suffer. I know things will improve as HTML5 gets wider adoption and we see more websites move away from Flash. In the meantime it is irritating although I’m sure Apple will make plenty of money from the games that will appear in the App store! All my 3 children liked it though, and it was amazing to see how quickly they picked up how to use it.

Last word, I will leave to Fake Steve in his open letter to the people of the world. Genius!