All posts by Neil Harwani

Interested in movies, music, history, computer science, software, engineering, management and technology

Learning R Programming – Part 1

As per Glassdoor Top 5 skills in Data Science for job openings are:

  1. Python
  2. R
  3. SQL
  4. Hadoop
  5. Java

Most Java developers know SQL, Hadoop & Java to a good extent in today’s environment, two important skills Python & R should be learned by the Java developer / architect / manager if s/he wants to contribute / work in Data Science area. In this article you will find a structured step by step approach for learning programming in R.

  1. To start with install R and familiarize yourself with R Console & R Script interface. You can run commands on both but it’s best to write multiple commands and try them out in script editor. Use short cut CTRL + R to run your commands in R Script editor.
  2. Explore menu options like Package -> Install / Load / Choose CRAN Mirror. By default many commands for statistics, visualization, etc. are given in R by default. Big set of libraries are already loaded into R by default and 100s more are available. Select any mirror to download new packages and install / load them step by step. You will need working internet connection. Learn how to set working directory. You can see default libraries available in R using library()
  3. From there move on to various R Objects / Data Types – Explore Data Frame, Vector, List, Matrices, Arrays and Factors. Try out examples for the same.
  4. Next step learn to load / read and write datasets by commands like read.csv / write.csv. You can also read / write excel sheets but for it you will need other packages. See the basic commands like summary, structure & fix to analyze / edit your dataset
  5. Next step – go through various categories of operators (logical, mathematical, relational) and concepts like pipe %>%, constants, rules for naming identifiers followed by various statistical functions directly available in R. You can get help on a command by using ?<COMMAND>. Also, learn to create functions and use conditions like if
  6. By now you should revise basics of statistics & various visualization charts which are taught typically in Year 1 / Semester 1 of MBA. Explore various default commands for statistics built into R by default. Some examples – mean, variance, standard deviation, etc.
  7. Learn to manipulate / read / write datasets using subset, sample_n & sample_frac and using dplyr package which has commands like select & filter among others
  8. Check various types of default visualization commands in R for various charts like barplot & pie. Post this learn how to use ggplot2 package
  9. You will get many datasets at kaggle.com and various websites like stock exchanges – NSE / BSE, RBI, Open Data websites of various Governments and others
  10. Explore top 20 packages of R categorized by various areas as given below.

An advantage of learning R is that you will become better at statistics & data science. It’s much simpler than Java in terms of syntax and structure and is influenced by open source languages / scripting like Linux, etc.

Reach out to me at neil@techandtrain.com if you want to discuss R, conduct a training for MBA / BE / MCA / MSc students in R or want to conduct a workshop for your managers / executives on Data Science / R / Java / etc.

References:

Top 10 skills for Data Science – Glassdoor Economic Research

Top 20 packages in R

Universities & Colleges offering courses & degrees in Analytics / Data Science in India – Part 1

  • BITS Pilani – Offering MSc in Analytics, MTech in Computer Science with specialization in Analytics
  • IIM Calcutta, ISI Kolkata & IIT Kharagpur – PG Diploma in Business Analytics
  • Great Lakes, Chennai – PGP in Analytics
  • IIIT Bangalore – PGP in Analytics via UpGrad
  • XLRI – Executive Program in Data Science
  • NIRMA University – MBA – Courses in 2nd year MBA covering Analytics & Data Science
  • Shanti Business School, Ahmedabad – PGDM having 2nd year specialization in Analytics / Decision Science
  • Gujarat University – MSc in Machine Learning & Artificial Intelligence
  • PG Diploma in Data Science – Manipal Global Academy in Data Science
  • Big Data & Visual Analytics – SP Jain School of Global Management
  • Master of Business Administration (Data Sciences and Data Analytics)- Symbiosis Center for Information Technology
  • Indian Institute of Information Technology and Management – Kerala (IITM-K) M.Sc. In Computer Science with Specialization in Data Analytics
  • Indian Institute of Science, Bangalore M.Tech. In Computational and Data Science & M.Tech.(Research) / PhD
  • Executive Program in Business Analytics- IIM Calcutta
  • St. Xavier’s College (Autonomous), Ahmedabad. (Collaboration with Tata Consultancy Services Limited) – M.Sc. in Big Data Analytics
  • IIIT Hyderabad – MTech in Data Science / Analytics

Not a complete list but gives you good data points on courses / degrees on Data Science / Analytics in India

Introduction to Search Engines – Part 1

Search engines have changed the way we work with the digital world completely. Digital world around us including the internet has evolved along with search engines almost simultaneously. Here is a primer on the working, technologies, statistics, history, tips for using, searching, etc. for Google.

How do they work?

Search engines work to search across billions of web pages, portals, documents, images, articles, news, journals, etc. to bring relevant and ranked results to your device. They do this by using few concepts like META Tags, Linking of web pages & content with each other, citations, source, web page credibility, social media relationships, configurations on websites like robots.txt / sitemap, etc.

Statistics

What you see as result on the first page of Google India is not the same in Google Europe for the same search word. Search results are customized and localized for each country, individual, session and even device at times. Currently as per publicly available statistics Google gets 40,000+ search requests per second which translates to few billion searches per day and would touch almost 1 Trillion searches approximately per year.

History of Google

Derived from word Googol and started at Stanford University. Initial algorithm was named BackRub.

Technology

Search engines have a spider / crawler which crawls across the internet to search and index / store the data in various databases. Ranking algorithm ranks all the content and refines it as per various parameters. Clustered servers across the world process the requests of search and provide content to devices. Lot of optimization based on new searches and corrections for old search terms / updating based on content changes happen regularly.

Tips & Tricks

Try: <vs.> <“EXACT SEARCH TERM”> <“SEARCH TERM -WITH SPECIFIC WORD EXCULED”> <“SITE: Search within a site”> <“DEFINE:”> <“SearchTerm1 * SearchTerm2 – Search everything including both terms and combinations with anything inbetween”>

Explore options of SETTINGS & TOOLS especially in IMAGE Search

Google Scholar

Search for scholarly papers, articles, theses, journals, books, reports, patents, etc.

Alternative search engines

Bing, DuckDuckGo, Yippy, Ask, DogPile, Yandex, etc.

If you want to learn more about the technologies used by search engines, try out:

Apache Solr

Apache Nutch

In Part – 2 we will discuss advanced topics like AdWords, AdSense, Google Analytics, Google Search Console, new implications of rankings based on https / SSL, details of Nutch & Solr, algorithms for search, other search engines like Yahoo, Bing, etc.

Facebook Privacy & Security tips

Many of us use Facebook for various reasons like socializing on the internet, managing and participating in groups, creating and managing company pages, keeping in touch with friends and colleagues, sharing our updates & pictures, etc. But many don’t pay attention to privacy and security aspects of using Facebook.

Here are some pointers on what areas of privacy & security you should be looking at in Facebook. This post does not give you what setting / configuration you should be keeping / selecting rather it makes you aware about those settings and you should select / configure the option based on your requirements.

  • Explore OTP / Two factor authentication using mobile / smartphone number / authentication app
  • Explore notifications over email & mobile
  • Explore options for your Cover & Profile picture – there is option to show your Profile picture to only your friends or network
  • Check whether your default posts are Public / shown to friends / shown to friends of friends / shown to friends except acquaintances
  • Check the type of groups you join: Public / Closed / Secret
  • Explore the privacy options like LIMIT YOUR OLD POSTS, Who can search you by email / phone, Who can see your friends list, etc.
  • Explore the options for people to Tag you in their posts
  • Explore the option to create custom username
  • Check the Location History option
  • Check which profiles in your network are Inactive / Spam / Without data / Suspect / Unknown. It’s best to remove them from your network by clicking UNFRIEND. Stay in touch with them over other mediums like Skype, LinkedIn, WhatsApp, SMS, Email, Signal, Google Hangout, etc.
  • Check the Apps & Website integration under Settings
  • Check the locations from where you are logged in under settings
  • Check your public posts by logging out & checking your profile or using the VIEW AS option under your profile
  • Check the setting whether search engines should link to your profile under privacy
  • Check the information in your profile added under ABOUT and who it is visible to on all sections and details of profile – Friends / Friends of Friends / Public / etc.
  • Check manage sections option in your profile and check privacy options for each of those sections
  • Check your advertisement preferences
  • Check the options which show your relationship with others under profile – whether it’s only shown to you, friends or friends of friends or public
  • Set up Trusted Contacts to recover your account
  • For your Facebook Mobile App check what permissions it has been given by the operating system (Android or iOS) under App permissions and review them based on your need
  • Check the people who are added as ADMIN for your company page if you manage one
  • There are a host of other options in settings for company pages too from which you can manage options like – from which country the visitors can see the page, whether tagging is allowed, age profile of your visitors, etc. If you are managing a company page you should explore them too

Security tips for Google Drive & Android

Google Drive:

  • Open Google Drive on web based browser like Chrome
  • On each of the folders if you see a person symbol it’s shared, if you don’t see the symbol its not shared
  • For each folder -> click SHARE -> Advanced -> Check which user has view rights and which user has edit rights -> Disable options to download, print, and copy for commenters and viewers -> Prevent editors from changing access and adding new people.
  • For each folder -> click SHARE -> Advanced -> SHARE SETTINGS -> Change -> Check settings there are 4 to 5 options
  • Note -> Even if your folder is not shared, files inside it can be in shared mode – This seems to be a big missing feature in Google Drive. There is no clear way to check this in one go for all files and we individually need to check each file. Workaround: Write Java code using Google Developers API to check this: https://developers.google.com/drive/

Android security settings to explore:

  • Emergency info – Settings -> Users -> Emergency info -> INFO & CONTACT – Note: Can be seen on locked screen
  • Settings -> Security -> Screen Lock
  • Encryption ON – at-least Android 6 required, by default it’s on from 6/7 onward – Settings -> Security
  • Settings -> Security -> Screen Lock -> Gear button -> Lock message
  • Security -> Install from unknown locations -> Should be OFF
  • Settings -> Users -> Guest user -> Turn on phone calls -> OFF
  • Settings -> Users -> Add users while screen is locked -> OFF
  • Settings -> Security -> Make password visible -> OFF
  • Settings -> Security -> Set up SIM card lock

How to build a lean startup in a Tier 2 or 3 city in India ? – Part 1

If you are exploring an option to start an IT / Digital startup in a Tier 2 or 3 city in India and are searching for some advice on how to do that in a lean, cost controlled way – here is some help:

  1. Register your company preferably in one of the three modes: Limited Liability Partnership, One Person Company OR Private Limited. There is a 4th option of proprietorship as well which is the easiest with less compliance requirements but with some other disadvantages. Select the one which suits your revenue, risks, appetite, customers and few other factors like in which line of business you are. You will need help of a CA and lawyer. Should include GST registration, Udyog Aadhaar, etc. Typical cost – Up to Rs. 20,000 plus invested amount for paid up capital – one time cost.
  2. You will need a business plan to chart out your path for at-least 1 to 2 years in the future. What will you offer: Products / Services / Or a mix of both? Where will you get the revenue from? Where will the funding come from? Who all will be your customers? What is the geography you will serve? Who will be your partners in management team? What partnerships in software / hardware with various companies do you need to have? How will you market & sell? What are the things you will outsource vs. do in-house? Where will the growth come from? Where are the future opportunities in the business space you want to serve? These are some basic questions that need to be answered in the business plan.
  3. Build your website preferably by investing time and taking help of another IT startup or SME. If you have the skills you can build it on your own. Typical cost for hosting & website, emails, some tools, cloud applications like project / requirement / salary management, storage, collaboration, CRM, etc. – up to Rs. 50,000 to 70,000 per year depending on what options, products, etc. you select.
  4. Register your Trade Mark. Typical cost – Rs. 10,000 to Rs. 50,000 depending on what all classes you file in and what are the further steps / objections. Once in 10 years.
  5. You will incur expense on office like properly lease, electricity & salaries along with maintenance of current account, debit cards, internet banking, etc. – Depends on the city and location along with employee strength – Typically Rs. 75,000 to 200,000 per month for 3 to 5 employee company in Tier 2 & 3 cities.
  6. You will need to meet legal & compliance requirements for filing taxes, annual meetings, audit, lawyer, etc. via a Chartered Accountant – Rs. 40,000 per year.
  7. Digital marketing & sales activities will be required to be done regularly. Here are some major ones: Advertisements on Google AdWords, Promotions on Facebook, Twitter, LinkedIn, etc. – Typically Rs. 50,000 to 75,000 per year.
  8. You will need to maintain Internet, computers and other associated administrative activities. Expenses – Rs. 20,000 per month.
  9. Setup and regular filing / maintenance for EPF / EPS, Employee medical / life / accident / professional liability insurance will need to be done. Related expenses – Depends on the size of your firm and revenue. But a reasonable amount for a small company of 10 employees is about Rs. 15,000 to 25,000 per month. Some of these are optional depending on the size of your company.

My suggestion is to go with a Tier 2 or 3 city as your base and use digital medium to offer your services along with usage of online products on taxation, legal, compliance, audit, salaries, collaboration, sales, marketing, etc. to manage your company as much as possible with as small a footprint as possible in physical world to start your venture.

You can save a lot on office lease, physical sales & meetings, fuel, meeting customers, collaboration and managing data if you go digital.

Reach out to me for any discussions, help, suggestions on building a lean and low cost startup. Good luck ! !

Skills required for working as a Solution Architect

  • High level Architecture understanding about Clustering of Databases / Applications Servers / Operating Systems, Enterprise Architecture, Non Functional Requirements, Integrations, Various Enterprise Software Products & Frameworks
  • Concepts of Virtualization and Cloud platforms like AWS, Azure & Google Cloud
  • Integration – Tools, Frameworks & Products that can be used like ESB, Web-Services, etc.
  • Latest trends around Big Data, NoSQL, React, Mobility, Portals, ERP, Cloud, Search, Scanning, Analytics, Networking, etc.
  • Reading list of websites for updating oneself on various domains, trends & technologies
  • Documents, spreadsheets & presentations – Skills with MS-Word, MS-Excel, MS-PPT, Google Docs, HTML, Visio, Free / Open Source / Cloud based Enterprise Architecture & Digital Marketing Tools & MS Paint, etc.
  • Programming skills in at-least one front line full application programming language like Java, VB.Net, C#, etc. as a minimum & concepts of digital electronics & microprocessors
  • Database, Encryption & Computer Networking skills including SQL, PL/SQL, Indexing, Triggers, Procedures, TCP/IP, UDP, IPv6/4, SSL/HTTPS, Types of encryption, etc.
  • Basics of how to create requirement documents, proposals, estimations & statement of works
  • Soft skills for working & collaborating with various stakeholders from Top Management, HR, Finance, Legal, Operations to developer level and customers
  • Reporting skills around creation of document, status updates, etc.
  • Project Management & Testing overview, Tools, Processes & Products used for the same
  • Overview of Quality Frameworks like CMMi, ISO, etc.
  • Ability to learn things quickly, present them in concise, customer friendly ways & networking skills to support Sales & Project Managers
  • Basic understanding of RFP / RFI / RFQ & Consulting process

Part 1 for this here: What do Presales Consultants / Solution Architects do ?

General Data Protection Regulation (GDPR) – Layman’s Introduction

  • Data of any EU citizen / entity cannot be shared without explicit consent. Terms & Conditions / Privacy checkbox doesn’t suitably give consent as per this law. Explicit consent is required
  • Records of when consent was given, how, what time, when revoked, etc. must be stored
  • Any breach of data or hacking must be informed to the stakeholders within 72 hours
  • It applies to whole of EU
  • Consent can be withdrawn anytime & when they want, and data processor / controller must comply
  • There is an option for right to be forgotten. If someone says to a data processor / controller that “FORGET ME” they must remove all data on them. It applies to all places which store & process / control the relevant data
  • GDPR applies to Production environment of Digital Systems which have data on EU citizens or entities. If Test / Pre-Production environment has unmasked data of EU citizens / entities again this law applies
  • https://www.zdnet.com/article/gdpr-an-executive-guide-to-what-you-need-to-know/ 

What should managers be doing in IT industry – Part 1

There are many things that senior engineers and managers do well in IT industry among which few are listed below:

  • Architecture
  • Presales
  • Solutioning
  • Debugging and solving problems
  • Exploring new technologies around the areas they have learned in the past
  • Analysis of scenarios
  • Product Management
  • Project Management
  • Setting up processes
  • Team Management
  • Digital Marketing & Sales

Some things that they don’t do well:

  • Put in long hours in support & maintenance
  • Explore completely new areas in which they have not worked before
  • Testing
  • Programming
  • Learning new trends and technologies

If you are born in 1980s or before, you would have become a manager in IT industry by now. In this case, at times many stop updating themselves.

Below are some pointers on how to update oneself and rebuild your career:

  • Instead of only reading about domains and technologies, experiment with software practically
  • Setup a small consulting / teaching / area specific business where you have passion, create a website, host some parts on traditional shared servers and some on AWS / Azure / etc.
  • Do some freelancing on the sides
  • Meet startups, founders, discuss their problems and offer free consulting hours to solve their issues
  • Network with local young IT professionals and see the latest apps and trends
  • Attend various MeetUps in your area
  • Pick up a new programming language and product
  • Try out Linux and other open source projects from SourceForge, GitHub, Apache Foundation, etc.
  • Take up courses from Coursera, edX, Udemy and similar portals
  • Go for further education in part time, online or distance mode – courses like MBA, MCA, Executive MBA, MSc, MS online from Universities across the world, MTech, PhD, etc. which can be done with your job
  • Follow industry professionals in upcoming technologies and domains on LinkedIn, Twitter and Facebook or other social media platforms to get updates

Unless one keeps up with the times they risk becoming obsolete !!

Advantages & Disadvantages of Crypto-Currencies

List of some of the major Crypto-Currencies as of now:

  • Bitcoin
  • Ripple
  • Ethereum
  • Litecoin
  • Zcash
  • Dash
  • Monero
  • More are on the way

Advantages of Crypto-Currencies:

  • Decentralized
  • Digital
  • Easy access
  • Traded on exchanges
  • Block-chain or similar modern database & protocol based non modifiable ledger of transactions
  • Immediate settlement and difficult to reverse / refuse payment
  • Lower fees
  • Difficult to do identity theft for conducting fraudulent transactions

Disadvantages of Crypto-Currencies:

  • Less or no regulation
  • No or less audits of exchanges where they trade
  • Price discovery does not follow standard financial rules
  • Mining is ad-hoc and affects money supply
  • Difficulty in reversal of payments in case of mistakes
  • Price volatility
  • Less clarity on taxation in some countries
  • Anonymity
  • Cutting edge technology which is yet to stabilize
  • Functionally features are complex for layman to understand