ipl-logo

Nt1310 Unit 3.4 Data Analysis

2265 Words10 Pages

3.4 Displaying meaningful results

Plotting points on a graph for analysis becomes difficult when dealing with extremely large amounts of information or a variety of categories of information. For example, imagine you have 10 billion rows of retail SKU data that you are trying to compare. The user trying to view 10 billion plots on the screen will have a hard time seeing so many data points. One way to resolve this is to cluster data into a higher-level view where smaller groups of data become visible. By grouping the data together, or “binning,” you can more effectively visualize the data.

3.5 Dealing with outliers

The graphical representations of data made possible by visualization can communicate trends and outliers much faster than tables …show more content…

HDFS runs across the nodes in a Hadoop cluster and together connects the file systems on many input and output data nodes to make them into one big file system. The present Hadoop ecosystem, as shown in Figure 1, consists of the Hadoop kernel, MapReduce, the Hadoop distributed file system (HDFS) and a number of related components such as Apache Hive, Hbase, Oozie, Pig and Zoo keeper and these components are explained as below …show more content…

Sqoop: A project for transferring/importing data between relational databases and Hadoop.

Oozie: An orchestration and work flow management for dependent Hadoop jobs.

Figure 2 gives an overview of the Big Data analysis tools which are used for efficient and precise data analysis and management jobs. The Big Data Analysis and management setup can be understood through the layered structured defined in the figure. The data storage part is dominated by the HDFS distributed file system architecture and other architectures available are Amazon Web Service, HBase and Cloud Store etc. The data processing tasks for all the tools is Map Reduce and it is the Data processing tool which effectively used in the Big Data Analysis[13].

For handling the velocity and heterogeneity of data, tools like Hive, Pig and Mahout are used which are parts of Hadoop and HDFS framework. It is interesting to note that for all the tools used, Hadoop over HDFS is the underlying architecture. Oozie and

EMR with Flume and Zoo keeper are used for handling the volume and veracity of data, which are standard Big Data management tools [13].

Figure 1 : Hadoop Architecture

Show More
Related

  • Nt1310 Unit 1 Data Analysis Essay

    395 Words  | 2 Pages

    SOF – The single dominant start of frame bit marks the start of a message, and is used to synchronize the nodes on a bus after being idle. Identifier-The Standard CAN 11-bit identifier establishes the priority of the message. Lower the value of identifier, its priority will be higher. RTR – The single remote transmission request (RTR) bit is used by the sender to inform receivers of the frame type.

    Read More

  • Nt1310 Unit 2 Research Abstract

    364 Words  | 2 Pages

    1.2 RESEARCH OBJECTIVES: 1.2.1 Creating Data Set The images used in this work have been obtained from the Street View service developed by Google. It provides high- resolution views from various positions along many streets and roads in the world. These images are taken at discrete geographical locations defined by a pair (LAT, LON) (latitude and longitude in decimal degrees, respectively) [19].

    Read More

  • Nt1310 Unit 4 Component Analysis

    331 Words  | 2 Pages

    Multilinear principal component analysis (MPCA) is a mathematical procedure that uses multiple orthogonal transformations to convert a set of multidimensional objects into another set of multidimensional objects of lower dimensions. There is one orthogonal (linear) transformation for each dimension (mode); hence multilinear. This transformation aims to capture as high a variance as possible, accounting for as much of the variability in the data as possible, subject to the constraint of mode-wise orthogonality. MPCA is a multilinear extension of principal component analysis (PCA).

    Read More

  • Nt1310 Unit 3 Lab Questions

    411 Words  | 2 Pages

    When it concerns the security of your home or business, sound locks are the first line of defense. However, it is an unfortunate fact that there are many people that are not particularly well-educated about locks. As a result, they may not be aware of the answers to a couple of common questions concerning these components of their security systems. After learning these two answers, you will be better able to minimize some of the problems that your locks might encounter.

    Read More

  • Pt1420 Unit 1 Assignment

    1121 Words  | 5 Pages

    1.1.2 Graphs We have now converted information from words or pictures to tables to formulae and now we’re going to look at how we can convert information into graphs: Example: If we invest R1 and it doubles every month, how much will we have at the end of 1 year? Let’s first draw a table: Months 1 2 3 4 5 6 7 8 9 10 11 12 Rands 1 2 4 8 16 32 64 128 256 512 1024 2048 Now let’s depict this information as a graph or chart: We can draw a bar chart:

    Read More

  • Nt1330 Unit 3 Data Analysis And Data Collection Case Study

    1070 Words  | 5 Pages

    Question 1: We learned that when calling subprograms or functions that we can pass data to the subprogram or function by ‘value’ or by ‘reference’. Describe what each approach is, how it works and what the potential security disadvantage is when passing parameters by reference. An easy way to view the difference between pass by value and pass by reference is to use the explanation on (Stack Overflow, 2008). This is a great way to visualize the difference as well as the potential security issues. "Say I want to share a web page with you.

    Read More

  • Nt1310 Unit 3 Quiz

    292 Words  | 2 Pages

    1. Are you aware of HAIs in your hospital? Yes or No 2. Which of the following of HAIs are commonly seen in your hospital  Urinary tract infections  Surgical wound infections  Respiratory tract infections  Bloodstream infection  Others, specify 3.

    Read More

  • Pt1420 Unit 3 Problem Analysis Paper

    269 Words  | 2 Pages

    Hadoop MapReduce is a framework for processing large data sets in parallel across Hadoop cluster. It uses two steps Map and Reduce process. An initiated job has a map and reduces phases. The map phase counts the words in each document, then reduce phase aggregates the per document data into word counts spanning the entire collection. The reduce phase use the results from map tasks as input to a set of parallel reduce tasks and it consolidate the data into final result.

    Read More

  • Nt1310 Unit 3 Quiz

    1644 Words  | 7 Pages

    1.8) Just as breathing and blinking involve a series of complex steps through complicated processes, our visual system has become effective and developed through evolution over millions of years. While the vision system may not be exactly performing this kind of mathematics, it has evolved in such a way that its design allows it to mimic this complex process. The gap between our visual systems abilities and our cognitive capabilities also helps free the conscious part of the mind, which allows animals to pay more attention to threats and be more likely to survive. 1.9) Evolution favors organisms that are able to live the longest and able to reproduce the most. Animals that acted irrationally were more likely to be killed and less likely to

    Read More

  • Nt1310 Unit 3 Research Methods

    617 Words  | 3 Pages

    n research methods, every researcher uses a procedure or a means of measurement to collect data. For example, three types of basic measurement collection are self-reports, observational, and physiological. Each method has their pros and cons in research. Depending on the research you are conducting these methods of measurement can either guide you to great discovery the pro, or skew your data making it unreliable the con. Observational measure is the method of measuring behaviors by directly observing subjects (Leary, M. R. (2011).

    Read More

  • AP Statistics Questions

    3183 Words  | 13 Pages

    Correlation isn’t resistant to extreme observations; outliers can have an affect on the

    Read More

  • ADI Research Assignment

    386 Words  | 2 Pages

    The t-chart

    Read More

  • Pieter Claesz Still Life Analysis

    1009 Words  | 5 Pages

    Analysis Many elements are visible in the drawing such as it’s usage of space, it is clear that the

    Read More

  • Predictive Analytics Essay

    942 Words  | 4 Pages

    As big data things continue to grow in this modern era, today we can learn how to predict or assume anything that will happen in the future with data from the past. This studies known as Predictive Analytics. Predictive analytics combine methods from machine learning, data mining and statistics to find meaning or pattern from a huge volume of data. Tom H Davenport, a senior advisor at Deloitte Analytics has broken down three primer models on doing predictive analytics: the data, statistics, and assumptions.

    Read More

  • Persuasive Essay On Information Technology

    949 Words  | 4 Pages

    Big Data There are many different definitions for Big Data. SAS (n.d.) an analytical software company describes it as, “a popular term used to describe the exponential growth and availability of data, both structured and unstructured.” Many think Big Data just came into existence but it has been around for years. Banks, retail, advertisers have been using big data for marketing purposes.

    Read More

More about Nt1310 Unit 3.4 Data Analysis

Open Document