{"id":573,"date":"2020-02-19T13:36:26","date_gmt":"2020-02-19T08:06:26","guid":{"rendered":"https:\/\/www.techandtrain.com\/blog\/?p=573"},"modified":"2025-10-22T14:54:59","modified_gmt":"2025-10-22T09:24:59","slug":"data-analysis-process-in-analytics-data-science","status":"publish","type":"post","link":"https:\/\/www.techandtrain.com\/blog\/2020\/02\/data-analysis-process-in-analytics-data-science\/","title":{"rendered":"Data Analysis Process in Analytics \/ Data Science"},"content":{"rendered":"\n

This article is based on understanding from Wikipedia article on Data Analysis & my experiences in Data Science \/ Analytics \/ AI \/ ML – <\/strong>https:\/\/en.wikipedia.org\/wiki\/Data_analysis<\/a><\/p>\n\n\n\n

Various areas like Data Mining, Predictive Analysis, Exploratory Data Analysis, Text Analytics, Business Intelligence, Confirmatory Data Analysis and Data Visualization<\/strong> overlap with this area<\/p>\n\n\n\n

Before starting your journey on solving an industry or academic or research problem in Data Science \/ Analytics \/ AI \/ ML \/ Decision Science, a fundamental step where many students & professionals struggle is Data Analysis<\/strong>. In this article, I am providing a step by step approach on analyzing your data. Directly starting with programming of various algorithms or neural network on your data could at times be counterproductive and should be avoided. <\/strong>Initial stage should involve robust data analysis<\/strong> via steps given below followed by model building which can include custom or already proven algorithms or a derivative of some popular models. For each of the points discussed below, I have added additional information on top of interpretation of Wikipedia information based on my experience in industry towards the end of each of the points or I have added new points post the interpretations.<\/strong><\/p>\n\n\n\n

Your steps for data analysis should generally be:<\/strong><\/p>\n\n\n\n

  1. Setup your data analysis process at a high level with your objectives – inspecting data, cleaning it, processing (could include\u00a0dimensionality reduction \/ feature engineering<\/strong>), transformation, modelling and communicating it.\u00a0Many forget the functional and feedback loop in this process setup to improve data quality – that must be included too.<\/strong><\/li>
  2. Next step is in understanding the data in terms of what is it telling us. Data could be quantitative style numbers or textual or a mix of it. Treatment for all three is different. For quantitative \/ numerical data, we try to understand whether it is\u00a0time-series, ranking, part to whole, deviation, frequency distribution, correlation, nominal or geographical or geospatial data<\/strong>. For textual or mixed type of data we need to use approaches of text mining, sentiment analysis, natural language processing to get insights around frequency of words, influential words & sentences by weight, trends, categories, clusters and more. Most of this article revolves around quantitative or numerical data perse and not textual data. I have provided a very brief idea on textual data analysis here in this point.<\/li>
  3. Next step would be to have the quantitative techniques being applied on the data in terms of sanity, audit \/ reconciliation of totals via\u00a0formulas, relationships between data, checking things like whether variables in data are related in terms of correlation \/ sufficiency \/ necessity<\/strong>\u00a0\/ etc. I would suggest using R Studio or similar tool for this step.<\/li>
  4. Post this we want to actually perform actions like\u00a0filtering, sorting, checking range and classes, summary, clusters, relationships, context, extremes,\u00a0<\/strong>etc. At this stage, exploratory data analysis techniques come in very handy where we use various libraries which provide graphical representation. Excel & Tableau come in handy here.<\/li>
  5. Our next step will be to check for\u00a0biases, deciphering facts & opinions, deciphering any numerical incorrect \/ irrelevant inferences<\/strong>\u00a0which are being projected and need correction \/ improvement. This needs detailed study of data from domain \/ functional perspective and applying statistical analysis on it. Working with a business \/ functional consultant in this phase is especially useful.<\/li>
  6. Some areas which we need to take care of include\u00a0quality of data, quality of measurements, transformations various variables \/ observations into log scale or others like what we have on richter scale for earthquakes, mapping to objectives and characteristics.\u00a0<\/strong>This is an intuitive step where visualizing data through various transformations in R \/ Python \/ etc. using libraries like Ggplot2, Plotly, Matplotlib, etc. helps.<\/li>
  7. Next comes\u00a0checking outliers, missing values, randomness, analysis & plotting various of charts<\/strong>\u00a0based on type of data whether categorical or continuous. This is statistical analysis & visualization where I find R to be most suited.<\/li>
  8. Building models around our data analysis steps could involve\u00a0linear, non-linear models and checking values via hypothesis testing and mapping to algorithms to process, predict, cluster, find trends<\/strong>\u00a0and so on. Products \/ tools like R \/ Python with libraries like Scikit learn, Numpy, Pandas, MLR, Caret, Keras, TensorFlow, etc. help here <\/li>
  9. While running the models take care of\u00a0cross-validation of data & sensitivity analysis<\/strong>\u00a0– This can generally be done using some options in model training & testing phase for supervised learning.<\/li>
  10. Feedback loop to circle and improve data & results, accuracy analysis and improvement, pipeline building, interpretation of results & functional mapping to domain<\/strong>\u00a0are additional things that we need to consider on top of the basics given in Wikipedia article. Also, things like dimensionality reduction techniques like PCA, SVD and such need to be explored in detail as they are helpful in this analysis.<\/li><\/ol>\n\n\n\n

    Additional information on top of what is in Wikipedia article:<\/strong><\/p>\n\n\n\n

    1. Explainable AI \/ ML – https:\/\/en.wikipedia.org\/wiki\/Explainable_artificial_intelligence<\/a><\/li>
    2. Interpretable ML – https:\/\/statmodeling.stat.columbia.edu\/2018\/10\/30\/explainable-ml-versus-interpretable-ml\/<\/a><\/li>
    3. Tools \/ languages \/ products to use: R, Python, Pandas, Numpy, Tableau and so on<\/li>
    4. EDA – https:\/\/en.wikipedia.org\/wiki\/Exploratory_data_analysis<\/a><\/li>
    5. Which chart to use – https:\/\/www.tableau.com\/learn\/whitepapers\/which-chart-or-graph-is-right-for-you<\/a><\/li>
    6. List of charts – https:\/\/python-graph-gallery.com\/all-charts\/<\/a><\/li>
    7. Confirmatory data analysis – https:\/\/en.wikipedia.org\/wiki\/Statistical_hypothesis_testing<\/a><\/li>
    8. Singular Value Decomposition – https:\/\/en.wikipedia.org\/wiki\/Singular_value_decomposition<\/a><\/li>
    9. Dimensionality Reduction – https:\/\/en.wikipedia.org\/wiki\/Dimensionality_reduction<\/a><\/li><\/ol>\n\n\n\n

      Email me: Neil@TechAndTrain.com<\/p>\n\n\n\n

      Visit my creations:<\/p>\n\n\n\n

      • www.TechAndTrain.com<\/li>
      • www.QandA.in<\/li>
      • www.TechTower.in<\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"

        This article is based on understanding from Wikipedia article on Data Analysis & my experiences in Data Science \/ Analytics […]<\/p>\n","protected":false},"author":2,"featured_media":574,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[72,82],"tags":[119,120,121],"class_list":["post-573","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-analytics","category-data-science","tag-dataanalysis","tag-dataanalytics","tag-dataexploration"],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2020\/02\/Geometry.jpg?fit=960%2C538&ssl=1","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7do02-9f","jetpack-related-posts":[{"id":557,"url":"https:\/\/www.techandtrain.com\/blog\/2020\/02\/what-are-we-doing-in-ai-ml-data-science-decision-science-analytics-world-glossary\/","url_meta":{"origin":573,"position":0},"title":"What are we doing in AI \/ ML \/ Data Science \/ Decision Science \/ Analytics World? – Glossary","author":"Neil Harwani","date":"February 17, 2020","format":false,"excerpt":"Over the last few years I have explored, programmed, worked in, researched and taught Data Science \/ AI \/ ML \/ Analytics \/ Decision Science to multiple students and with many software professionals. I have collected many keywords that you can google and explore. This will help you to keep\u2026","rel":"","context":"In "Analytics"","block_context":{"text":"Analytics","link":"https:\/\/www.techandtrain.com\/blog\/category\/analytics\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2020\/02\/Mathematics.jpg?fit=960%2C490&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2020\/02\/Mathematics.jpg?fit=960%2C490&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2020\/02\/Mathematics.jpg?fit=960%2C490&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2020\/02\/Mathematics.jpg?fit=960%2C490&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":417,"url":"https:\/\/www.techandtrain.com\/blog\/2019\/02\/what-should-be-the-subjects-course-structure-for-teaching-data-analytics-data-science-in-mba\/","url_meta":{"origin":573,"position":1},"title":"What should be the subjects & course structure for teaching Data Analytics \/ Data Science in MBA?","author":"Neil Harwani","date":"February 12, 2019","format":false,"excerpt":"Data Science & Analytics including Operations \/ Decision Science are evolving fields which are in demand currently for various reasons. Most companies are experimenting and creating projects \/ products around analytics \/ data science. I am listing the subjects & courses that an MBA student should take to cover Data\u2026","rel":"","context":"In "Analytics"","block_context":{"text":"Analytics","link":"https:\/\/www.techandtrain.com\/blog\/category\/analytics\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2019\/02\/Analytics2.jpg?fit=960%2C635&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2019\/02\/Analytics2.jpg?fit=960%2C635&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2019\/02\/Analytics2.jpg?fit=960%2C635&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2019\/02\/Analytics2.jpg?fit=960%2C635&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":698,"url":"https:\/\/www.techandtrain.com\/blog\/2021\/03\/four-waves-of-artificial-intelligence-machine-learning\/","url_meta":{"origin":573,"position":2},"title":"Four waves of Artificial Intelligence & Machine Learning","author":"Neil Harwani","date":"March 19, 2021","format":false,"excerpt":"While teaching students in two different courses (AIML & \"Data Science and Analysis\"), there was a requirement to categorize historical AI & ML along with it's interface with Data Science. To start: AI is the superset, ML is a subset of AI, Neural Networks (Deep Learning) are specialized subsets of\u2026","rel":"","context":"In "Analytics"","block_context":{"text":"Analytics","link":"https:\/\/www.techandtrain.com\/blog\/category\/analytics\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2021\/03\/LighBulb.jpg?fit=1200%2C675&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2021\/03\/LighBulb.jpg?fit=1200%2C675&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2021\/03\/LighBulb.jpg?fit=1200%2C675&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2021\/03\/LighBulb.jpg?fit=1200%2C675&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2021\/03\/LighBulb.jpg?fit=1200%2C675&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":336,"url":"https:\/\/www.techandtrain.com\/blog\/2018\/08\/universities-colleges-offering-courses-degrees-in-analytics-data-science-in-india-part-1\/","url_meta":{"origin":573,"position":3},"title":"Universities & Colleges offering courses & degrees in Analytics \/ Data Science in India – Part 1","author":"Neil Harwani","date":"August 14, 2018","format":false,"excerpt":"BITS Pilani - Offering MSc in Analytics, MTech in Computer Science with specialization in Analytics IIM Calcutta, ISI Kolkata & IIT Kharagpur - PG Diploma in Business Analytics Great Lakes, Chennai - PGP in Analytics IIIT Bangalore - PGP in Analytics via UpGrad XLRI - Executive Program in Data Science\u2026","rel":"","context":"In "Analytics"","block_context":{"text":"Analytics","link":"https:\/\/www.techandtrain.com\/blog\/category\/analytics\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2018\/08\/Analytics-3.png?fit=1200%2C1047&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2018\/08\/Analytics-3.png?fit=1200%2C1047&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2018\/08\/Analytics-3.png?fit=1200%2C1047&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2018\/08\/Analytics-3.png?fit=1200%2C1047&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2018\/08\/Analytics-3.png?fit=1200%2C1047&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":367,"url":"https:\/\/www.techandtrain.com\/blog\/2018\/10\/how-to-explore-and-learn-analytics-data-science\/","url_meta":{"origin":573,"position":4},"title":"How to explore and learn “Analytics & Data Science” ?","author":"Neil Harwani","date":"October 10, 2018","format":false,"excerpt":"One of my students asked me as to how\u00a0can someone\u00a0explore and learn Analytics \/ Data Science domain with an intention to build their career in it ? There are three types of roles available in Data Science \/ Analytics: Functional consultant like a Business \/ Data Analyst Technical Consultant like\u2026","rel":"","context":"In "Analytics"","block_context":{"text":"Analytics","link":"https:\/\/www.techandtrain.com\/blog\/category\/analytics\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2018\/10\/Analysis-2.jpg?fit=1200%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2018\/10\/Analysis-2.jpg?fit=1200%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2018\/10\/Analysis-2.jpg?fit=1200%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2018\/10\/Analysis-2.jpg?fit=1200%2C800&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2018\/10\/Analysis-2.jpg?fit=1200%2C800&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":1288,"url":"https:\/\/www.techandtrain.com\/blog\/2024\/08\/learnings-from-emasters-data-science-for-decision-making-iit-gandhinagar-part-1\/","url_meta":{"origin":573,"position":5},"title":"Learnings from eMasters Data Science for Decision Making – IIT Gandhinagar – Part 1","author":"Neil Harwani","date":"August 11, 2024","format":false,"excerpt":"Below are the learnings from eMasters Data Science for Decision Making - IIT Gandhinagar - Part 1. This needs to be an article in itself. You can find more details here about eDSDM here: e-Masters | IITGNX There is Mathematics for your AIML problem in feature engineering, preprocessing, evaluation metrics,\u2026","rel":"","context":"In "Analytics"","block_context":{"text":"Analytics","link":"https:\/\/www.techandtrain.com\/blog\/category\/analytics\/"},"img":{"alt_text":"Credits: www.Pixabay.com","src":"https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2024\/08\/DataScience.jpg?fit=1200%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2024\/08\/DataScience.jpg?fit=1200%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2024\/08\/DataScience.jpg?fit=1200%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2024\/08\/DataScience.jpg?fit=1200%2C800&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.techandtrain.com\/blog\/wp-content\/uploads\/2024\/08\/DataScience.jpg?fit=1200%2C800&ssl=1&resize=1050%2C600 3x"},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.techandtrain.com\/blog\/wp-json\/wp\/v2\/posts\/573","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.techandtrain.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.techandtrain.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.techandtrain.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.techandtrain.com\/blog\/wp-json\/wp\/v2\/comments?post=573"}],"version-history":[{"count":4,"href":"https:\/\/www.techandtrain.com\/blog\/wp-json\/wp\/v2\/posts\/573\/revisions"}],"predecessor-version":[{"id":578,"href":"https:\/\/www.techandtrain.com\/blog\/wp-json\/wp\/v2\/posts\/573\/revisions\/578"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.techandtrain.com\/blog\/wp-json\/wp\/v2\/media\/574"}],"wp:attachment":[{"href":"https:\/\/www.techandtrain.com\/blog\/wp-json\/wp\/v2\/media?parent=573"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.techandtrain.com\/blog\/wp-json\/wp\/v2\/categories?post=573"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.techandtrain.com\/blog\/wp-json\/wp\/v2\/tags?post=573"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}