Planning consideration for a big data infrastructure

 

Big data is here. it’s no secret that the flood of information from internet, sensors, logs and images holds untapped opportunities for organisations looking to improve the bottom line. Data and business analysts have made it clear that machine generated data is essential to future success of their organisations. These data exit in large volumes and the reality the data streams at high data rate than what IT systems typically handle today. By providing access to broader sets of information, big data can help by maximising data analysts. Successful big data analytics uncover trends and pattens that enable business user to make better decision to new revenue opportunities and keep the organisations a head of their business ravels. But first, the organisation often need  to enhance their existing IT infrastructure and data management process including data governance to support the scale and complexity of a big data architecture.

Embracing new technologies are always challenging. Expanding data centre to include big data, IT and data governance must ensure business alignment, establish standards and manage best practices. Data infrastructure architect is expected to provide a fast and reliable path to business adoption.

There are common considerations that must be addressed to ensure an effective framework for big data efforts. The following considerations that needs to taken into account:-

 

Scalability

Big data platform should be built with the future in mind. It’s imperative to incorporate the whether the technology being evaluated to scale to the level of the business requirements. To that extends, storage capacity, computing power and performance are crucial ingredient for big data scalability. The new architecture for big data should be able to cop with capacity and performance. Non-stop data growth and the need for speed continue to be the driving force behind for housing, managing and protecting big data.

 

Integration

Data integrations are growing fast. Organisations are demanding that they can change directions and requirements so that they can get insight gain from various sources of data. As result, the Big Data architecture design should provide data integration solution as as a service to use of different stands of new source of data. Before moving ahead on a Big Data integration initiative, organisation should consider how much data they need and how much of what they have can be used to produce real business value. This task should be in the Data Architect remits. SOA and ESB are must ingredient to address the challenge of data integration.

Scoping

Basic business rule for capitalising on Big Data, the business users and data architect need to define the value of Big Data for the organisation. Inaccurate scope is a recipe for the Big Data project failure. To avoid the inaccurate scope pitfall, the organisation needs to ensure the Big Data project is scope by considering the following:

Business use cases

The organisation needs to understand the broader use cases for Big Data and identify which use case the organisation is trying to address in term of priority.

Data Source

The Data Architect identifies what data source needs to support the target business cases so that an architecture development can start.

Design

Following establishing the information architecture at the Data Source stage, the design part of Big Data starts. This includes specifying hardware, operating system, storage, network and analytics and application stacks.

Cataloguing

The Proof of Concept (PoC) should start to go through the system design and all the processes been set in place to ensure the data sources can integrate the require data in the Big Data platform. Following the integration, the business users apply the analytics to the data set.

Quality

Data quality and cleansing are the norm practice of the Business Intelligence team. The Business Intelligence and analytics teams strive to ensure the validity of data and convince the business users to trust the data in term of accuracy and reliability of information assets. By providing standardised processes for managing Business Intelligence and analytics data – utilising data quality tools, can help the business user confident. But a Big Data implementation adds a degree of difficulties in term of increase data volumes and wide area of variety of data types, particular when a mix of structure and unstructured data is involved. The approach to assess data quality is vital to the successful implementation and usage of a Big Data analytics framework.

 We focus on business intelligence and Big Data deployments at small and mid-size businesses. We have over 15+ years experience in data analysis, system analysis, database design, infrastructure design and implementation of enterprise applications.

 

If you need further information or initial consultancy, please email mustafa@imexservices.co.uk for details.