Batch processing has traditionally been used in many organizations to perform non real-time, data intensive tasks that are executed at specific frequencies and within predefined time periods.  For example, loading bulk customer data into a Customer database, creating summary records for a data warehouse, etc.  The processing and performance requirements of these batch processes tend to differ from those of real-time, interactive processes.  In order to address these specific requirements, organizations have invested in creating custom, homegrown batch processing solutions and this resulted in a heterogeneous mix of technologies with no standardization.  Applying the concepts of Service Oriented Architecture to batch processing enables organizations to not only standardize on implementation technologies/approaches but also leverages the many benefits offered by an enterprise wide SOA.

As Ronald Schmelzer of ZapThink puts it aptly in his note,

Service-Oriented Architecture (SOA) presents enterprises with the opportunity to expose information and processes as self-contained Services that can communicate and interoperate with each other in a standard, loosely coupled fashion. Although the common impression is that Services expose business processes, data processes, or application functionality, they are also well-suited for exposing the very processes that drive batch-oriented workloads. SOA enables the business to build flexible compositions of Services that implement either business or IT processes in a loosely coupled manner, which has important ramifications for IT service delivery, and the batch processes that are part of it.

However, there seems to be a common misconception in many enterprises regarding the applicability of service orientation to batch processes.  I have heard at least twice in the past week from someone that batch processes do not really lend themselves well to Service-Oriented Architecture (SOA) and that SOA is best suited for building dynamic, interactive, transactional applications.  While batch processing remains a critical component in many data intensive environments in today’s real-time enterprises for various reasons, I think the role and potential applications of batch processes has changed with the advent of SOA.  Batch processes are evolving from static, stand-alone processes invoked at specific frequencies and time windows to essential, dynamic components of business driven process flows that can be invoked on-demand.

The following are some of the ways batch processes can be first class citizens in enterprise wide service oriented architecture.

• Batch Processes as Service Providers
  • Batch processes can be built and exposed as reusable services to be consumed on-demand by other batch and/or real-time business processes and services.
• Batch Processes as Service Consumers
  • Batch processes can consume other enterprise services that provide business and/or utility functions as part of the core processing logic.  For example, an ETL process that is bulk loading new customer data into an operational database can use a service that returns a unique customer ID for each customer from an authoritative data source.
• Batch Processes as Policy–driven, Configurable Work Flows
  • Modern batch processes can be designed, managed and executed very similar to their real-time counter parts using configurable process flows and meta data to control the behavior, frequency, time window and sequence of execution.
• Event Driven invocation of batch processes
  • Batch processes do not need to be invoked on a fixed schedule by job schedulers and cron jobs anymore – they can be part of  a dynamic, event-driven SOA by responding to business events in addition to scheduled timer events.  For example, the same batch process that creates summary records in a data warehouse can be invoked in response to a real-time change by the business users or in response to the arrival of  a new batch feed.
• Separation of business logic, rules as reusable components
  • Traditionally the processing logic and rules are embedded in batch processes with very little reuse across the processes, let alone across the enterprise.  There is tremendous value in extracting the business rules and reusable business logic out of the batch processes and exposing them as reusable components or services at the enterprise level so that real-time and batch processes alike can utilize the same components or services irrespective of the mode of invocation.

However, introducing batch processes into SOA has its limitations and challenges.  Not all batch processes may be the best candidates to become service providers or consume services as part of their processing logic.  The biggest potential downside of applying service orientation to batch processes is decrease in performance.  Batch processes have historically processed large volumes of data in pre-defined time windows overnight and introducing service calls into these processes could affect performance.  Hence, the benefits of applying SOA to batch processes such as increased reuse and increased agility must be carefully weighed against the potential implications on performance.  However, these concerns can be addressed and mitigated by laying a solid technical foundation that meets the rigorous performance requirements of batch processes and by utilizing work load automation frameworks that enable an organization adapt to changing processing needs without affecting the core functionality of batch processes.  For example, batch frameworks such as Spring Batch, written in Java, provide the required abstraction to introduce fault tolerance and sclability into batch processes for high volume processing.  In addition, depending on the performance requirements of an organization, other advanced options such as integration with grid computing solutions to partition the batch job over a large number of processors can be considered to ensure high speed processing.

As I mentioned earlier, batch processing is a critical part of standard business operations for many organizations and that will not change any time soon.  As organizations move toward modern architectures from legacy systems and processes, batch processing must be looked at from a new perspective and must be treated as an integral part of SOA and Enterprise Architecture in order to realize the true potential of an integrated, agile enterprise.

Visual Thinking

Visual thinking is our brain’s natural way of solving problems by picturing the problem and it’s various outcomes visually in our mind.  It enables us to abstract out the complexity in a complex problem or information and present the same in a much simpler and more meaningful way that is easy to comprehend.  Going back to the map example, it’s far easier and natural for us to visualize the route from point A to point B instead of remembering the textual directions in a list form.  We use visual thinking consciously or unconsciously quite often in our daily life – whether it is a sketch of an idea we drew up in a coffee shop or a brainstorming session using a white board or a more fancy presentation for the board.   In business world, power point presentations with fancy graphics, excel spreadsheets with charts and executive dashboards with all kinds of visual gadgets remain the most popular way to present complex business concepts and information.  Here is an excellent example of using visual thinking to describe the concept of Twitter.

Another excellent source of information on visual thinking is this video of David Armano, VP of Experience Design with Critical Mass, speaking at a conference on thinking visually.  According to David Armano, thinking visually may one of the most sought after abilities of the 12st century.

Effective communication is everyone’s job—whether you are trying to sell in a concept or convince a client. Visual Thinking can help us take in complex information and synthesize it into something meaningful. In an increasingly fragmented and cluttered world, simple imagery, metaphors and mindmaps can get people to understand the abstract and make your ideas tangible.

Click here to see some excellent visual thinking artifacts from David Armano.

Mind Maps

A mind map uses visual thinking to create a visual representation of the problem and the possible solutions in a diagram that mirrors the way our mind naturally processes information.  It is a diagram used to represent words, ideas, tasks, or other items linked to and arranged around a central key word or idea. Mind maps are used to generate, visualize, structure, and classify ideas, and as an aid in study, organization, problem solving, decision making, and writing.  By presenting ideas in a radial, graphical, non-linear manner, mind maps encourage a brainstorming approach to planning and organizational tasks (Source: Wikipedia).

For example, if we wanted to draw a mind map for capturing all the activities surrounding starting a business – we can start with a central idea called “Start a Business”.  Radiating from this central idea, we can add branches for “Business Objectives”, “Business Plan”, “Market Research”, “Project Plan”, etc.  Each of these branches will now become the central idea for the creation it it’s own mind map thus enabling us to create a high level view and a detail view of what’s involved in starting our business.

Start a New Business Mind Map - Drawn using MindMeister

Mind maps can be used to capture and solve complex problems in personal life or in business – for example, you can use a mind map to capture ideas during a brainstorming session or take notes during a lecture or create a non-liner to-do list, etc.  Mind maps enable us to start with a bird’s eye view of the problem and drill down into specific areas that require further exploration.  In addition, each area or branch within a mind map can be enhanced by attaching an artifact that is relevant to that branch — for example, a spread sheet with budget numbers or an email received from a customer, etc.  Mind maps can be drawn by hand or by using any of the several commercial and open source software available in the market.  Here are a some resources to explore the concept of mind maps a bit more:

• Mind Mapping Software Blog, By Chuck Frey
• The Mind map Blog, by Chance Brown
• Idea Mapping Blog, by Jamie Nast
• The MindJet Blog: 20+ Mind Mapping Resources, By MindJet
• 30+ Mind Mapping Tools, By Mashable

Visual Modeling for Complex Business Problems

Idiagram's Approach (© copyright 2002-2007 Marshall Clemens)

Visual modeling is also used to model and solve complex business problems that are non-technical such as, business strategy, vision, business models, communication models, knowledge integration, etc.   During one of my recent online explorations about visual modeling,  I ran into a company called Idiagram owned by Mr. Marshall Clemens, that uses visual modeling based approach to solve complex business problems such as strategic planning & decision making, product innovation & design, dialog facilitation, knowledge management, creative thinking and marketing strategy.  I was very impressed with Idiagram’s approach and the way Mr.Clemens described the art of complex problem solving.  I also loved the fact that Mr.Clemens used visual models and his own technique to describe all concepts on the company’s web site — which not only makes it very easy to understand but also provides a great example of using visual models to abstract and simplify complexity in any domain.

Mr.Clemens does not define what a “visual model” should look like — instead uses any diagram or style (diagrams, maps, charts, pictures, etc.) that represents the concept under consideration effectively. Here are a few samples of Mr.Clemens’s work and more samples of his work can be found here.

Idiagram's Process (© copyright 2002-2007 Marshall Clemens)

Graphic Facilitation (© copyright 2002-2007 Marshall Clemens)

Knowledge Integration (© copyright 2002-2007 Marshall Clemens)

In essence, irrespective of the target domain and the potential application of visual modeling within that domain, the underlying premise of using visual models remains the same — minimize complexity, visualize concepts, facilitate communication and increase collaboration to realize a common goal.

Modeling is an inherent part of human life — as they say “a picture is worth a thousand words”.  A picture can communicate information better by hiding the complexities and displaying only the relevant details.  For example, when we look for directions we tend use the map representation instead of the textual directions as the map provides us with just enough details to visualize the directions.  Models also have been used in several disciplines including science and engineering for a long time.   CAD/CAM models are used in engineering to model engineering parts and structural models are used in architecture to model large structures and buildings.  Irrespective of the problem domain, models provide a simplified abstraction of the complexity involved in that domain.  While models do not have to be visual, most models are visual and the visual representation further adds to minimization of complexity.  Different models can be used to capture different characteristics of the problem domain and the type of diagrams used to visualize the model of a specific problem domain really depends on the problem domain itself.

Visual models have also been used in software engineering to different extents depending on the context, skills, time and tools available.  Depending on the size of the organization or team working on project and the complexity involved, no modeling step may be involved in certain implementations.  Developers go directly from requirements to coding and no models are produced during the lifecycle of the project.  In other cases, models drawn on white board are used to facilitate discussions around design ideas and modeling stops when the discussion ends.  In most cases, models are used primarily for communication among various groups involved during the early stages of the project – during requirements gathering, analysis and design.  The primary intent of these models is to capture a high level view of the system under consideration and to present a common view to all parties involved.  These models could be technical architectural diagrams, data models, use case models, class diagrams, sequence diagrams, etc., and are drawn using tools such as Powerpoint or Visio or a more formal UML modeling tool.  However, these models are thrown away once the development starts since it is hard to keep the code and models in sync as the requirements change during the remaining life cycle of the project.

Today’s systems are complex with many moving parts (thanks to modern multi-tier and distributed architectures) — models enable us to cope with this complexity by providing a visual abstraction layer that focuses on the higher level concepts in the problem domain and de-couple the “what” from “how”.  For instance, imagine trying to understand the dependencies in a large system by looking at the code vs. looking at visual design models (use case models, component diagrams, class diagrams and activity diagrams, etc.).  Depending on the level of detail desired, the needs of the target stake holder and the choice of architectural framework (TOGAF, Zachman, etc.), multiple views or perspectives of the entire enterprise architecture can be captured in various models including use cases, system architecture, data flows, object models, business processes, physical network, deployment nodes, system dependencies and database schemas, etc.

In summary, visual models provide us with the following important benefits:

• Minimize complexity by capturing complex moving parts in multiple views, enable clear overall view of interactions
• Enable effective communication among stake holders, business analysts, enterprise architects and software developers by providing a common view, language and notation
• Facilitate collaboration via model sharing, transition and refinement between domain experts (business modeling), architects (system modeling) and software developers (analysis, design and implementation)
• Increase developer productivity and shorten the development life cycle by providing ways to generate executables from models either at design time (code generation) or run time (executable models).

As I mentioned earlier, in most traditional code-based development approaches, these models are merely used as design artifacts that provide a starting point for an extensive coding exercise where the higher level details captured in these models are translated into lower level details based on the target architecture.  What if the models we created are semantically 100% complete and are immediately executable instead of merely being design artifacts?  This is the promise of Model Driven Development and more on “Executable Models” later.