As an increasing number of organizations gear up to implement an efficient and effective Big Data system to help them improve performance and deliver better results, most encounter certain roadblocks they weren’t prepared for. It is, therefore, imperative that you understand what exactly you are getting into before you jump on to the Big Data bandwagon.

What are the considerations?

There are a quite a number of factors you need to keep in mind while planning the transition from tracking traditional transactional data to the rich and unstructured data formats that come in from the web.

• Budget: Most traditional data servers weren’t built to process large volumes of data. Big Data, however, requires analytic and high performance computing servers, which would entail considerable IT investment.
• IT Know-how: Big Data requires a different strategy for data storage and processing. It needs a tiered storage to place the data at different locations depending on the how frequently they are retrieved. So,you will have to formulate your own rules for prioritizing and accessing your Big Data.
• Business Know-how: It is easy to get a solution based on the reports you’ve been using in the past. Big Data, however, promises much more, but only if you have the right resources who could query it to find the relevant answers.
• Market competition: More focus on personalization in Real Time is crucial for retaining hold in Market.

What’s the way forward?

Although there’s no singular way of deploying such a business solution to cater to industry-specific needs, a planned approach would go a long wayin minimizing risk and increasing the probability of a successful outcome.

• Identify Stakeholders: Usually, the Big Data stakeholders are knowledge workers and decision makers. You will have to categorize them by role, prioritize their decision-making value, and pursue a sequential and progressive roadmap.
• Identify Data Stewards: You would need to find the right people to define data governance and implement the data management process.
• Qualify Data: Since the system would rely considerably on the data available, the database will have to be cleaned-up to ensure that all inaccurate, duplicate and incomplete data is removed. This data would then have to be segregated to retain all that is important and archive the rest.
• Prepare for the Transition: Introducing Big Data would change the work dynamics. The increase in the velocity, volume, variety of data might mean change in work hours/timings for real-time data analysis. In addition, new policies will have to be formulated to determine what would happen to the data after the expiry of retention timeframes.
• Create the Plan & Establish Metrics: While planning the transition you must keep in mind that Big Data is complementary to and not a replacement for your business analytics. So, limit the metrics to a few high priority ones rather than having an exhaustive list.
• Deploy the Technology: Get the tools, software and other resources required to implement Big Data onboard. Given the fact that we have numerous tools and technologies, implementing aProof-Of-Concept(POC) before final design is always recommended
• Align Business and IT Needs: You need to know what you intend to accomplish by implementing Big Data and sync your IT strategies accordingly to achieve business excellence.

What are the challenges?

Although most of us know that information harnessed from external and internal sources is valuable, it is more important to make it accessible and actionable by knowledge workers whenever and wherever required. Also, more data means more noise. So, business analysts would have to classify and filter the data received. Issues pertaining to data privacy, information distribution, information security, data presentation etc. will also need to be addressed.

What are the best practices?

• Know your business needs before you start collecting the data
• The use case needs to be well defined. Not every problem is a big data problem
• Clearly identify the approach, Batch or Real time for the use case

• Use an agile and interactive approach for implementation
• Start small with incremental steps and then scale up
• Choose each component wisely before starting with implementation

• Evaluate your data requirements
• Choose the component wisely before starting with implementation

• Promote knowledge transfer

• Encourage experimentation and subsequent re-configuration
• Do not discard any data, even if you have completed your analysis on it
• Find different uses for the same data

• Align the system with the cloud operating model

• Associate big data with organizational data
• Make the data available to multiple teams to run analytics and or visualization on top of it
• Connect your Big Data infra to already existing Data Warehouse and other systems

• Use business intelligence to embed analytics and decision-making into operational workflows

Read complete blog