After writing lots of blogs on events and meet-up, now its time for some change. So, this time I am writing a blog on a very Big topic i.e. Big Data.
After learning about ‘Big Data’ and its concepts I personally think the name should have been ‘Vig Data’ because there are lots of V’s involved here. (That was a joke 🙂 )
So before you go through this long time killing blog, Just want to tell you that the purpose of writing this blog is to give you an initial knowledge about Big Data in a summarized way.
Definitions of Big Data:
- Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.
- Big data is a popular term used to describe the exponential growth and availability of data, both structured and unstructured.
- Every day, we create 2.5 quintillion bytes of data — so much that 90% of the data in the world today has been created in the last two years alone. This data is big data.
- The term big data, especially when used by vendors, may refer to the technology (which includes tools and processes) that an organization requires to handle the large amounts of data and storage facilities.
- Big Data usually includes data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process the data within a tolerable elapsed time.
- Big data is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.
The Origin of Big Data :
- Big Data Originated as the three Vs: Volume, Velocity, and Variety.
- This is the most venerable and well-known definition, first coined by Doug Laney of Gartner over twelve years ago.
- Since then, many others have tried to take it to eleven with additional Vs including Validity, Veracity, Value, and Visibility etc.
Why Big Data?:
You’ll Manage Data Better:
Many of today’s big data and business intelligence tools let users sit in the driver’s seat and work with data without going through too many complicated technical steps.
This added layer of abstraction has enabled numerous use cases where data in a wide variety of formats has been successfully mined for specific purposes.
- One example is real-time video processing.
End Users Can Visualize Data:
A big data initiative is going to require next-level data visualization tools, which present Big data in easy-to-read charts, graphs and slide-shows.
- In the vast quantities of data being examined, these applications will be able to offer processing engines that let end users query and manipulate information quickly.
New Business Opportunities:
As big data analytics tools continue to mature, more users are realizing the competitive advantage to being a data-driven enterprise.
Social media sites have identified opportunities to generate revenue from the data they collect by selling ads based on an individual user’s interests. This lets companies target specific sets of individuals that fit an ideal client or prospect profile.
- Big data use cases in about in retail, where the focus is on gaining insights by studying consumer behaviour in online stores or physical shopping centres.
Your Data Analysis Methods, Capabilities Will Evolve:
Data is no longer simply numbers in a database. Text, audio and video files can also provide valuable insight; the right tools can even recognize specific patterns based on predefined criteria.
This happens using natural language processing tools, which can prove vital to text mining, sentiment analysis, clinical language and name entity recognition efforts.
- One example that highlights the use of audio analysis and big data comes from Matter-sight. This call centre tool can match incoming caller to the appropriate customer agent by using predictive behavioural routing and other analytics technology.
Big Data allows ever-narrower segmentation of customers and therefore much more precisely tailored products or services.
Sophisticated analytics can substantially improve decision-making, minimize risks, and unearth valuable insights that would otherwise remain hidden.
Big Data can unlock significant value by making information transparent.
- Big Data can be used to develop the next generation of products and services.
So how does it work?
- It depends on the technology used and what you are trying to achieve through the use of BigData.
- BigData consists of different types of technologies which work together to achieve the end goal: extracting value from data that would have been previously considered ‘dead’.
- Here are some of the key technologies / concepts associated with BigData:
Hadoop, HDFS, NoSQL, MapReduce, MongoDB, Cassandra, PIG, HIVE, HBase.
Hadoop and Big Data:
- Doug Cutting, Cloudera’s Chief Architect, helped create Apache Hadoop out of necessity as data from the web exploded, and grew far beyond the ability of traditional systems to handle it.
- Hadoop was initially inspired by papers published by Google outlining its approach to handling an avalanche of data, and has since become the de facto standard for storing, processing and analyzing hundreds of terabytes, and even petabytes of data.
- Apache Hadoop is 100% open source, and pioneered a fundamentally new way of storing and processing data.
- Instead of relying on expensive, proprietary hardware and different systems to store and process data, Hadoop enables distributed parallel processing of huge amounts of data.
- With Hadoop, no data is too big. And in today’s hyper-connected world where more and more data is being created every day, Hadoop’s breakthrough advantages mean that businesses and organizations can now find value in data that was recently considered useless.
Reveal Insight From All Types of Data, From All Types of Systems
Hadoop can handle all types of data from disparate systems: structured, unstructured, log files, pictures, audio files, communications records, email– just about anything you can think of, regardless of its native format.
You don’t need to know how you intend to query your data before you store it. Hadoop lets you decide later and over time can reveal questions you never even thought to ask.
- Hadoop lets you see relationships that were hidden before and reveal answers that have always been just out of reach.
Redefine the Economics of Data:
Hadoop’s cost advantages over legacy systems redefine the economics of data.
Legacy systems, while fine for certain workloads, simply were not engineered with the needs of Big Data in mind and are far too expensive to be used for general purpose with today’s largest data sets.
As it relies in an internally redundant data structure and is deployed on industry standard servers rather than expensive specialized data storage systems, you can afford to store data not previously viable.
- Enterprises who build their Big Data can afford to store literally all the data in their organization, and keep it all online for real-time interactive querying, business intelligence, analysis and visualization.
Restructure Your Thinking:
- With data growing so rapidly and the rise of unstructured data accounting for 90% of the data today, the time has come for enterprises to re-evaluate their approach to data storage, management and analytics.
- Legacy systems will remain necessary for specific high-value, low-volume workloads, and compliment the use of Hadoop-optimizing the data management structure in your organization by putting the right Big Data workloads in the right systems.
- The cost-effectiveness, scalability and streamlined architectures of Hadoop will make the technology more and more attractive.
Apache Hadoop is an open-source software framework for storage and large-scale processing of data-sets on clusters of commodity hardware. Hadoop is an Apache top-level project being built and used by a global community of contributors and users.It is licensed under the Apache License 2.0.
The Apache Hadoop framework is composed of the following modules:
- Hadoop Common – contains libraries and utilities needed by other Hadoop modules
- Hadoop Distributed File System (HDFS) – a distributed file-system that stores data on commodity machines, providing very high aggregate bandwidth across the cluster.
- Hadoop YARN – a resource-management platform responsible for managing compute resources in clusters and using them for scheduling of users’ applications.
- Hadoop MapReduce – a programming model for large scale data processing.
Apache Hadoop’s MapReduce and HDFS components originally derived respectively from Google’s MapReduce and Google File System (GFS) papers.
Hello!! If you are still reading this blog, Just want to thank you for your patience.
If you don’t like reading too much of text then we have something for you too.
In my next blog we will discuss “How to setup up single-node Hadoop cluster backed by the Hadoop Distributed File System, running on Ubuntu Linux. ”
Hope you enjoyed reading my blog.
Have a great time!
Terms And Conditions Of Use:
All content provided on this blog (http://wp.me/p2IUwr-gn) is for informational purposes only. The owner of this blog makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site.
The owner(https://fosswithumesh.wordpress.com/) of this blog will not be liable for any errors or omissions in this information nor for the availability of this information. The owner will not be liable for any losses, injuries, or damages from the display or use of this information.