Ashok Nare's Blog

Batch Processing in a Service-Oriented Architecture

Posted at August 12, 2009

Batch processing has traditionally been used in many organizations to perform non real-time, data intensive tasks that are executed at specific frequencies and within predefined time periods. For example, loading bulk customer data into a Customer database, creating summary records for a data warehouse, etc. The processing and performance requirements of these batch processes tend to differ from those of real-time, interactive processes. In order to address these specific requirements, organizations have invested in creating custom, homegrown batch processing solutions and this resulted in a heterogeneous mix of technologies with no standardization. Applying the concepts of Service Oriented Architecture to batch processing enables organizations to not only standardize on implementation technologies/approaches but also leverages the many benefits offered by an enterprise wide SOA.

As Ronald Schmelzer of ZapThink puts it aptly in his note,

Service-Oriented Architecture (SOA) presents enterprises with the opportunity to expose information and processes as self-contained Services that can communicate and interoperate with each other in a standard, loosely coupled fashion. Although the common impression is that Services expose business processes, data processes, or application functionality, they are also well-suited for exposing the very processes that drive batch-oriented workloads. SOA enables the business to build flexible compositions of Services that implement either business or IT processes in a loosely coupled manner, which has important ramifications for IT service delivery, and the batch processes that are part of it.

However, there seems to be a common misconception in many enterprises regarding the applicability of service orientation to batch processes. I have heard at least twice in the past week from someone that batch processes do not really lend themselves well to Service-Oriented Architecture (SOA) and that SOA is best suited for building dynamic, interactive, transactional applications. While batch processing remains a critical component in many data intensive environments in today’s real-time enterprises for various reasons, I think the role and potential applications of batch processes has changed with the advent of SOA. Batch processes are evolving from static, stand-alone processes invoked at specific frequencies and time windows to essential, dynamic components of business driven process flows that can be invoked on-demand.

The following are some of the ways batch processes can be first class citizens in enterprise wide service oriented architecture.

Batch Processes as Service Providers
- Batch processes can be built and exposed as reusable services to be consumed on-demand by other batch and/or real-time business processes and services.
Batch Processes as Service Consumers
- Batch processes can consume other enterprise services that provide business and/or utility functions as part of the core processing logic. For example, an ETL process that is bulk loading new customer data into an operational database can use a service that returns a unique customer ID for each customer from an authoritative data source.
Batch Processes as Policy–driven, Configurable Work Flows
- Modern batch processes can be designed, managed and executed very similar to their real-time counter parts using configurable process flows and meta data to control the behavior, frequency, time window and sequence of execution.
Event Driven invocation of batch processes
- Batch processes do not need to be invoked on a fixed schedule by job schedulers and cron jobs anymore – they can be part of a dynamic, event-driven SOA by responding to business events in addition to scheduled timer events. For example, the same batch process that creates summary records in a data warehouse can be invoked in response to a real-time change by the business users or in response to the arrival of a new batch feed.
Separation of business logic, rules as reusable components
- Traditionally the processing logic and rules are embedded in batch processes with very little reuse across the processes, let alone across the enterprise. There is tremendous value in extracting the business rules and reusable business logic out of the batch processes and exposing them as reusable components or services at the enterprise level so that real-time and batch processes alike can utilize the same components or services irrespective of the mode of invocation.

However, introducing batch processes into SOA has its limitations and challenges. Not all batch processes may be the best candidates to become service providers or consume services as part of their processing logic. The biggest potential downside of applying service orientation to batch processes is decrease in performance. Batch processes have historically processed large volumes of data in pre-defined time windows overnight and introducing service calls into these processes could affect performance. Hence, the benefits of applying SOA to batch processes such as increased reuse and increased agility must be carefully weighed against the potential implications on performance. However, these concerns can be addressed and mitigated by laying a solid technical foundation that meets the rigorous performance requirements of batch processes and by utilizing work load automation frameworks that enable an organization adapt to changing processing needs without affecting the core functionality of batch processes. For example, batch frameworks such as Spring Batch, written in Java, provide the required abstraction to introduce fault tolerance and sclability into batch processes for high volume processing. In addition, depending on the performance requirements of an organization, other advanced options such as integration with grid computing solutions to partition the batch job over a large number of processors can be considered to ensure high speed processing.

As I mentioned earlier, batch processing is a critical part of standard business operations for many organizations and that will not change any time soon. As organizations move toward modern architectures from legacy systems and processes, batch processing must be looked at from a new perspective and must be treated as an integral part of SOA and Enterprise Architecture in order to realize the true potential of an integrated, agile enterprise.

Comments

Why Do We Need Visual Models ? (Part 2)

Posted at April 5, 2009

In my previous post “Why Do We Need Visual Models ?” I talked about how visual models help simplify complexity in technical problems and various applications of visual models in software development. In this post, I would like to explore alternate uses of visual models and the application of visual modeling to solve complex non-technical problems in business and life.

Visual Thinking

Visual thinking is our brain’s natural way of solving problems by picturing the problem and it’s various outcomes visually in our mind. It enables us to abstract out the complexity in a complex problem or information and present the same in a much simpler and more meaningful way that is easy to comprehend. Going back to the map example, it’s far easier and natural for us to visualize the route from point A to point B instead of remembering the textual directions in a list form. We use visual thinking consciously or unconsciously quite often in our daily life – whether it is a sketch of an idea we drew up in a coffee shop or a brainstorming session using a white board or a more fancy presentation for the board. In business world, power point presentations with fancy graphics, excel spreadsheets with charts and executive dashboards with all kinds of visual gadgets remain the most popular way to present complex business concepts and information. Here is an excellent example of using visual thinking to describe the concept of Twitter.

Another excellent source of information on visual thinking is this video of David Armano, VP of Experience Design with Critical Mass, speaking at a conference on thinking visually. According to David Armano, thinking visually may one of the most sought after abilities of the 12st century.

Effective communication is everyone’s job—whether you are trying to sell in a concept or convince a client. Visual Thinking can help us take in complex information and synthesize it into something meaningful. In an increasingly fragmented and cluttered world, simple imagery, metaphors and mindmaps can get people to understand the abstract and make your ideas tangible.

Click here to see some excellent visual thinking artifacts from David Armano.

Mind Maps

A mind map uses visual thinking to create a visual representation of the problem and the possible solutions in a diagram that mirrors the way our mind naturally processes information. It is a diagram used to represent words, ideas, tasks, or other items linked to and arranged around a central key word or idea. Mind maps are used to generate, visualize, structure, and classify ideas, and as an aid in study, organization, problem solving, decision making, and writing. By presenting ideas in a radial, graphical, non-linear manner, mind maps encourage a brainstorming approach to planning and organizational tasks (Source: Wikipedia).

For example, if we wanted to draw a mind map for capturing all the activities surrounding starting a business – we can start with a central idea called “Start a Business”. Radiating from this central idea, we can add branches for “Business Objectives”, “Business Plan”, “Market Research”, “Project Plan”, etc. Each of these branches will now become the central idea for the creation it it’s own mind map thus enabling us to create a high level view and a detail view of what’s involved in starting our business.

Start a New Business Mind Map - Drawn using MindMeister

Mind maps can be used to capture and solve complex problems in personal life or in business – for example, you can use a mind map to capture ideas during a brainstorming session or take notes during a lecture or create a non-liner to-do list, etc. Mind maps enable us to start with a bird’s eye view of the problem and drill down into specific areas that require further exploration. In addition, each area or branch within a mind map can be enhanced by attaching an artifact that is relevant to that branch — for example, a spread sheet with budget numbers or an email received from a customer, etc. Mind maps can be drawn by hand or by using any of the several commercial and open source software available in the market. Here are a some resources to explore the concept of mind maps a bit more:

Mind Mapping Software Blog, By Chuck Frey
The Mind map Blog, by Chance Brown
Idea Mapping Blog, by Jamie Nast
The MindJet Blog: 20+ Mind Mapping Resources, By MindJet
30+ Mind Mapping Tools, By Mashable

Visual Modeling for Complex Business Problems

Visual modeling is also used to model and solve complex business problems that are non-technical such as, business strategy, vision, business models, communication models, knowledge integration, etc. During one of my recent online explorations about visual modeling, I ran into a company called Idiagram owned by Mr. Marshall Clemens, that uses visual modeling based approach to solve complex business problems such as strategic planning & decision making, product innovation & design, dialog facilitation, knowledge management, creative thinking and marketing strategy. I was very impressed with Idiagram’s approach and the way Mr.Clemens described the art of complex problem solving. I also loved the fact that Mr.Clemens used visual models and his own technique to describe all concepts on the company’s web site — which not only makes it very easy to understand but also provides a great example of using visual models to abstract and simplify complexity in any domain.

Mr.Clemens does not define what a “visual model” should look like — instead uses any diagram or style (diagrams, maps, charts, pictures, etc.) that represents the concept under consideration effectively. Here are a few samples of Mr.Clemens’s work and more samples of his work can be found here.

Idiagrams Process (copyright 2005 Marshall Clemens)

In essence, irrespective of the target domain and the potential application of visual modeling within that domain, the underlying premise of using visual models remains the same — minimize complexity, visualize concepts, facilitate communication and increase collaboration to realize a common goal.

Modeling, Technical, Visual Modeling

Comments

Networking For First Time Entrepreneurs

Posted at March 30, 2009

Effective networking is an extremely important tool for first time entrepreneurs. Irrespective of target goals – meeting potential investors, finding business partners, establishing technology partnerships, or simply spreading the word around about your new venture – networking provides the necessary channels to meet those goals if utilized properly.

Robert Ochtel offers some good tips on “dos and don’ts” of networking for first time entrepreneurs in his blog post titled ‘Some Truths About Networking for First Time Entrepreneurs‘. I especially like his comment on how networking is really focused “Socializing”:

“Networking is really no different than any other socializing activity. In fact, if it were referred to as “socializing”, instead of networking, I believe it would be less intimidating to first time entrepreneurs. The word “networking” seems to have an underlying performance-based stigma associated with it………A better approach is to attend each networking event with a positive attitude and hope to meet one to two individuals you can possibly create a personal connection with.”

It’s true that people are far more relaxed in social settings (with friends and family or casual gatherings) than in networking events. As Robert points out, this is due to “performance based stigma” associated with networking events that creates an expectation of walking away with contacts that are potentially valuable for you or your venture. In this case, the concept of “value” is essentially misplaced. Networking rarely generates the type of contacts and relationships that create an immediately impact on your venture overnight — for example, you cannot meet an investor and expect him to invest in your company immediately. Instead, focus must be on cultivating mutually beneficial, long term relationships that might potentially add value to you and your venture in the long term.

Following up with the contacts you meet at any event within a week or so of that event is also important step in building a relationship with that contact. Networking does not end when you meet a person at a conference or an event – but continues way beyond the event. When you meet someone at a networking event, you are just someone they “met at an event” – like many others they might have met that day at that particular event. Following up and maintaining a continuous dialogue will give you an opportunity to distinguish yourself from the pack and stand out.

Here are some “do’s and don’ts” of following up and keeping an open ended conversation going:

Do send an email within a week of meeting the contact, giving them the context of your meeting such as date and place of the event, any specific conversation you had with the contact, etc.
Do follow the blog of the contact if there is one and contribute to the conversation on the blog thro’ comments.
Do follow the person on Twitter – this is effective and non-intrusive way of keeping up with updates of the contact.
Do Not send a LinkedIn connection request immediately — they don’t know you well enough yet to be on their list of connections (this may not apply to someone who is an open networker).
Do Not send a Facebook connection request immediately — again, they don’t know you well enough for you to be on their list of friends. Most people also use Facebook exclusively for managing their personal network (and LinkedIn for professional) only and it might be a long time before you can connect with them on Facebook at that level.

To expand on the “targeted networking” Robert talked about i his post, here are a few more tips on how to network effectively.

Give before you receive: Key to developing effective long term network is to be willing to give without an expectation of receiving. Share knowledge and resources with the people in your network and all that “good karma” will come back to you eventually when you need help from your network.
Contribute to blogs: Take part in the conversations on the blogs you follow and contribute comments, feedback and resources back to the community.
Volunteer: Volunteer for a not-for-profit organization that is a better fit for you and your venture in your target industry. In addition to giving you an opportunity to give back to the world, it also gives you an excellent channel for networking.
Join professional associations:Joining a professional association that matches your (or your venture’s) area of expertise is also an effective way to be part of a community of people with similar interests.
Attend local meetup events: Find and attend a meetup event in your local area. Depending on the topic and area of interest, there are several meetups happening in every city, every day. Try a few in your area until you find the ones that fit your interests and profile.
Organize networking events: Another sure fire way to meet new people is to organize a networking event yourself. These events can be as simple as a happy hour in your favorite restaurant or a meetup in a local coffee shop or even a breakfast event organized by you or your company.

Networking is a tough but crucial component of the first time entrepreneur’s overall strategy for personal and professional success. I myself am practicing most of the tips from above and from Robert’s post — and several others are already on my networking road map.

Got any more tips on networking for first time entrepreneurs ? Share them here. In the mean time, get out there and start networking !