Batch Processing in a Service-Oriented Architecture

Batch Processing in SOA

Batch processing has traditionally been used in many organizations to perform non real-time, data intensive tasks that are executed at specific frequencies and within predefined time periods.  For example, loading bulk customer data into a Customer database, creating summary records for a data warehouse, etc.  The processing and performance requirements of these batch processes tend to differ from those of real-time, interactive processes.  In order to address these specific requirements, organizations have invested in creating custom, homegrown batch processing solutions and this resulted in a heterogeneous mix of technologies with no standardization.  Applying the concepts of Service Oriented Architecture to batch processing enables organizations to not only standardize on implementation technologies/approaches but also leverages the many benefits offered by an enterprise wide SOA.

As Ronald Schmelzer of ZapThink puts it aptly in his note,

Service-Oriented Architecture (SOA) presents enterprises with the opportunity to expose information and processes as self-contained Services that can communicate and interoperate with each other in a standard, loosely coupled fashion. Although the common impression is that Services expose business processes, data processes, or application functionality, they are also well-suited for exposing the very processes that drive batch-oriented workloads. SOA enables the business to build flexible compositions of Services that implement either business or IT processes in a loosely coupled manner, which has important ramifications for IT service delivery, and the batch processes that are part of it.

However, there seems to be a common misconception in many enterprises regarding the applicability of service orientation to batch processes.  I have heard at least twice in the past week from someone that batch processes do not really lend themselves well to Service-Oriented Architecture (SOA) and that SOA is best suited for building dynamic, interactive, transactional applications.  While batch processing remains a critical component in many data intensive environments in today’s real-time enterprises for various reasons, I think the role and potential applications of batch processes has changed with the advent of SOA.  Batch processes are evolving from static, stand-alone processes invoked at specific frequencies and time windows to essential, dynamic components of business driven process flows that can be invoked on-demand.

The following are some of the ways batch processes can be first class citizens in enterprise wide service oriented architecture.

However, introducing batch processes into SOA has its limitations and challenges.  Not all batch processes may be the best candidates to become service providers or consume services as part of their processing logic.  The biggest potential downside of applying service orientation to batch processes is decrease in performance.  Batch processes have historically processed large volumes of data in pre-defined time windows overnight and introducing service calls into these processes could affect performance.  Hence, the benefits of applying SOA to batch processes such as increased reuse and increased agility must be carefully weighed against the potential implications on performance.  However, these concerns can be addressed and mitigated by laying a solid technical foundation that meets the rigorous performance requirements of batch processes and by utilizing work load automation frameworks that enable an organization adapt to changing processing needs without affecting the core functionality of batch processes.  For example, batch frameworks such as Spring Batch, written in Java, provide the required abstraction to introduce fault tolerance and sclability into batch processes for high volume processing.  In addition, depending on the performance requirements of an organization, other advanced options such as integration with grid computing solutions to partition the batch job over a large number of processors can be considered to ensure high speed processing.

As I mentioned earlier, batch processing is a critical part of standard business operations for many organizations and that will not change any time soon.  As organizations move toward modern architectures from legacy systems and processes, batch processing must be looked at from a new perspective and must be treated as an integral part of SOA and Enterprise Architecture in order to realize the true potential of an integrated, agile enterprise.

Reblog this post [with Zemanta]
Service Oriented Architecture, Technical

Why Do We Need Visual Models ? (Part 2)

In my previous post “Why Do We Need Visual Models ?” I talked about how visual models help simplify complexity in technical problems and various applications of visual models in software development.  In this post, I would like to explore alternate uses of visual models and the application of visual modeling to solve complex non-technical problems in business and life.

Visual Thinking

Visual thinking is our brain’s natural way of solving problems by picturing the problem and it’s various outcomes visually in our mind.  It enables us to abstract out the complexity in a complex problem or information and present the same in a much simpler and more meaningful way that is easy to comprehend.  Going back to the map example, it’s far easier and natural for us to visualize the route from point A to point B instead of remembering the textual directions in a list form.  We use visual thinking consciously or unconsciously quite often in our daily life – whether it is a sketch of an idea we drew up in a coffee shop or a brainstorming session using a white board or a more fancy presentation for the board.   In business world, power point presentations with fancy graphics, excel spreadsheets with charts and executive dashboards with all kinds of visual gadgets remain the most popular way to present complex business concepts and information.  Here is an excellent example of using visual thinking to describe the concept of Twitter.

Another excellent source of information on visual thinking is this video of David Armano, VP of Experience Design with Critical Mass, speaking at a conference on thinking visually.  According to David Armano, thinking visually may one of the most sought after abilities of the 12st century.

Effective communication is everyone’s job—whether you are trying to sell in a concept or convince a client. Visual Thinking can help us take in complex information and synthesize it into something meaningful. In an increasingly fragmented and cluttered world, simple imagery, metaphors and mindmaps can get people to understand the abstract and make your ideas tangible.

Click here to see some excellent visual thinking artifacts from David Armano.

Mind Maps

Start a New Business Mind Map - Hand DrawnA mind map uses visual thinking to create a visual representation of the problem and the possible solutions in a diagram that mirrors the way our mind naturally processes information.  It is a diagram used to represent words, ideas, tasks, or other items linked to and arranged around a central key word or idea. Mind maps are used to generate, visualize, structure, and classify ideas, and as an aid in study, organization, problem solving, decision making, and writing.  By presenting ideas in a radial, graphical, non-linear manner, mind maps encourage a brainstorming approach to planning and organizational tasks (Source: Wikipedia).

For example, if we wanted to draw a mind map for capturing all the activities surrounding starting a business – we can start with a central idea called “Start a Business”.  Radiating from this central idea, we can add branches for “Business Objectives”, “Business Plan”, “Market Research”, “Project Plan”, etc.  Each of these branches will now become the central idea for the creation it it’s own mind map thus enabling us to create a high level view and a detail view of what’s involved in starting our business.

Start a New Business Mind Map - Drawn using MindMeister

Start a New Business Mind Map - Drawn using MindMeister

Mind maps can be used to capture and solve complex problems in personal life or in business – for example, you can use a mind map to capture ideas during a brainstorming session or take notes during a lecture or create a non-liner to-do list, etc.  Mind maps enable us to start with a bird’s eye view of the problem and drill down into specific areas that require further exploration.  In addition, each area or branch within a mind map can be enhanced by attaching an artifact that is relevant to that branch — for example, a spread sheet with budget numbers or an email received from a customer, etc.  Mind maps can be drawn by hand or by using any of the several commercial and open source software available in the market.  Here are a some resources to explore the concept of mind maps a bit more:

Visual Modeling for Complex Business Problems

Idiagram's Approach (© copyright 2002-2007 Marshall Clemens)

Visual modeling is also used to model and solve complex business problems that are non-technical such as, business strategy, vision, business models, communication models, knowledge integration, etc.   During one of my recent online explorations about visual modeling,  I ran into a company called Idiagram owned by Mr. Marshall Clemens, that uses visual modeling based approach to solve complex business problems such as strategic planning & decision making, product innovation & design, dialog facilitation, knowledge management, creative thinking and marketing strategy.  I was very impressed with Idiagram’s approach and the way Mr.Clemens described the art of complex problem solving.  I also loved the fact that Mr.Clemens used visual models and his own technique to describe all concepts on the company’s web site — which not only makes it very easy to understand but also provides a great example of using visual models to abstract and simplify complexity in any domain.

Mr.Clemens does not define what a “visual model” should look like — instead uses any diagram or style (diagrams, maps, charts, pictures, etc.) that represents the concept under consideration effectively. Here are a few samples of Mr.Clemens’s work and more samples of his work can be found here.

Idiagrams Process (copyright 2005 Marshall Clemens)

Idiagram's Process (© copyright 2002-2007 Marshall Clemens)

Graphic Facilitation ()

Graphic Facilitation (© copyright 2002-2007 Marshall Clemens)

Knowledge Integration ()

Knowledge Integration (© copyright 2002-2007 Marshall Clemens)

In essence, irrespective of the target domain and the potential application of visual modeling within that domain, the underlying premise of using visual models remains the same — minimize complexity, visualize concepts, facilitate communication and increase collaboration to realize a common goal.

Enhanced by Zemanta
Modeling, Technical, Visual Modeling

Networking For First Time Entrepreneurs

Effective networking is an extremely important tool for first time entrepreneurs.  Irrespective of target goals – meeting potential investors, finding business partners, establishing technology partnerships, or simply spreading the word around about your new venture – networking provides the necessary channels to meet those goals if utilized properly.

Robert Ochtel offers some good tips on “dos and don’ts” of networking for first time entrepreneurs in his blog post titled ‘Some Truths About Networking for First Time Entrepreneurs‘.  I especially like his comment on how networking is really focused “Socializing”:

“Networking is really no different than any other socializing activity. In fact, if it were referred to as “socializing”, instead of networking, I believe it would be less intimidating to first time entrepreneurs. The word “networking” seems to have an underlying performance-based stigma associated with it………A better approach is to attend each networking event with a positive attitude and hope to meet one to two individuals you can possibly create a personal connection with.”

It’s true that people are far more relaxed in social settings (with friends and family or casual gatherings) than in networking events.  As Robert points out, this is due to “performance based stigma” associated with networking events that creates an expectation of walking away with contacts that are potentially valuable for you or your venture.  In this case, the concept of “value” is essentially misplaced.  Networking rarely generates the type of contacts and relationships that create an immediately impact on your venture overnight — for example, you cannot meet an investor and expect him to invest in your company immediately.  Instead, focus must be on cultivating mutually beneficial, long term relationships that might potentially add value to you and your venture in the long term.

Following up with the contacts you meet at any event within a week or so of that event is also important step in building a relationship with that contact.  Networking does not end when you meet a person at a conference or an event – but continues way beyond the event.  When you meet someone at a networking event, you are just someone they “met at an event” – like many others they might have met that day at that particular event.  Following up  and maintaining a continuous dialogue will give you an opportunity to distinguish yourself from the pack and stand out.

Here are some “do’s and don’ts”  of following up and keeping an open ended conversation going:

To expand on the “targeted networking” Robert talked about i his post, here are a few more tips on how to network effectively.

Networking is a tough but crucial component of the first time entrepreneur’s overall strategy for personal and professional success.  I myself am practicing most of the tips from above and from Robert’s post — and several others are already on my networking road map.

Got any more tips on networking for first time entrepreneurs ?  Share them here.  In the mean time, get out there and start networking !

Enhanced by Zemanta
Entrepreneurship, Networking