Learnings from eMasters Data Science for Decision Making – IIT Gandhinagar – Part 1

Below are the learnings from eMasters Data Science for Decision Making – IIT Gandhinagar – Part 1. This needs to be an article in itself. You can find more details here about eDSDM here: e-Masters | IITGNX

  • There is Mathematics for your AIML problem in feature engineering, preprocessing, evaluation metrics, models, errors, etc. and so on or you can build some relationship between Mathematics and your problem either by breaking the problem into pieces or transforming data and so on.
  • Human intuition is still invaluable like in situations of imbalanced datasets, regression and more.
  • Visualization and EDA almost always help for your problems. For higher dimension problems you can do PCA, T-SNE, shadowing on lower dimensions, etc. more approaches to bring it to lower dimensions with or without transformation like basis transformation. In this situation, you should also feed data after this transformation for testing / using the model.
  • There are problems beyond model selection and preprocessing, feature engineering like data quality, overfitting, evaluation, errors, hyper parameter tuning and generalization which we need to think about.
  • Situations like classification of medical diagnosis issues have very high value repercussions for wrong classification whereas some models with overfitting won’t generalize and cause problems like in normal situations with regression.
  • Probability concepts like equally likely for randomness, distributions, fairness, probability tree, joint probability, permutations and combinations, etc. have lot of value and are worth learning.
  • There is no learning like learning by experience, concepts and application.
  • Your assumptions maybe wrong as well so it’s good to verify.
  • Accuracy alone may not be a good measure, please add recall & precision as well in your analysis especially in imbalanced datasets.
  • Domain knowledge matters. Don’t ignore this.
  • Website: www.HarwaniSystems.in
  • Blog: www.TechAndTrain.com/blog
  • LinkedIn: Neil Harwani | LinkedIn
  • Email: Neil@HarwaniSystems.in

Returning to school / academics from industry

Below is an article summarizing some points that I have experienced when transitioning back to academics / school from industry. I have done academics (learning) part time which is study part time since 2011 onwards to achieve various goals in academics along with work in industry. Here is the summary of what it takes and some tips to excel:

  • You need to accept that getting a degree or a good certification takes time and effort.
  • You need dedicated time over the nights or mornings on weekdays and especially half of the weekends sacrificing time with family and friends.
  • Calendaring or scheduling time using calendars is your best friend.
  • Finding out the best resources from the internet and Wikipedia or similar portals is very helpful.
  • You can do anything but not everything. This is actually true. You need to drop / deprioritize what you cannot do due to lack of time.
  • Your industry & family environment needs to be supportive of your goals and efforts, only then you will be able to manage both industry and academics.
  • Writing / journaling also definitely helps, something like a blog as well can help.
  • Pick growth mindset, have an open mind and learn continuously. This needs to become a habit.
  • Use your industry knowledge to have discussions with batchmates/peers and professors. This helps to learn quickly and have engaging discussions.
  • Integrate industry practices into your academic work.
  • Network with industry, professors, batchmates to learn more effectively and stay on top of trends.
  • Take advantage of academic assets like libraries and online databases for research.
  • Use online platforms for research, collaboration, and project management.
  • Understand mixing theory and practice. It helps.
  • Aim to bridge the gap between academia and industry in your work.
  • Plan your savings and finances to manage academic expenses properly.
  • Maintain a healthy balance between work, life and study time.
  • Manage stress through exercise, proper nutrition, and mindfulness practices.
  • Set clear, achievable, planned goals and not ad-hoc random expectations. Adjust as necessary.
  • Be flexible to new academic environments.
  • Try for innovation using your industry and academic knowledge.
  • Returning to good academic degrees / diplomas / certifications / workshops will most likely improve your knowledge and skills significantly.
  • Website: www.HarwaniSystems.in
  • Blog: www.TechAndTrain.com/blog
  • LinkedIn: Neil Harwani | LinkedIn
  • Email: Neil@HarwaniSystems.in

Mastering Data Series – Enterprise Content Management – Part 1

Here are the main points for Enterprise Content Management in Mastering Data series – Part 1 from architecture & technology perspective:

  1. Hierarchical object model to store content & documents with it’s attributes (extended and / or default) in the core engine
  2. Publishing module for content release to audience
  3. Portal for accessing these content
  4. Use cases: Workflow over content, Content publishing, Digital Asset Management, Digital Rights Management, Scanning Solutions, Search, Content Sharing, Content Automation, Knowledge Management, Knowledge Discovery, Insights/Analytics over documents and so on
  5. Scanning solutions: OCR, ICR, OMR, HCR, Barcode, Watermarking and so on
  6. Compliance / Retention / Governance
  7. Lifecycle of ECM: Capture, Manage, Store, Preserve & Deliver
  8. Integrations: SoA, Pub-Sub, ESB, Asynchronous integration and so on
  9. Transformation services between various formats
  10. Distribution over email, paper, internet, etc.
  11. What all comes in the deployment typically: Search, File Store, Core Engine, Workflow Engine, Integration module, Portal, Scanning Solution, Publishing Module, Social Media, AI/ML Insights, Data Science over documents module, etc.
Enterprise content management – Wikipedia

Pointers to work with product support at Liferay – Part 1

Below pointers may help when working with Liferay support to cut response times:

1. How is the vanilla product behaving for your problem area?

2. What are the relevant database tables, source code and configurations saying? Discuss about this early on the ticket

3. Attach any video of the problem that you might have?

4. Have you checked whether it’s truly a Liferay issue or an issue with the environment and ecosystem around it like web server, CDN, WAF and so on. These details should also go on the ticket.

5. Are your logs verbose enough? If not, did you enable detailed logging via control panel.

6. Have you scanned logs of Liferay, Elastic Search, Database, Web server, etc. What errors are being thrown?

7. Did you check thread & heap dump, CPU, Memory, etc.? It will help in certain related situations.

8. Do you have Glowroot enabled? What is it saying about errors, slow traces, JVM, etc.?

9. Are all systems online like File Store, Elastic Search, Database, Web Server, WAF, CDN, etc.

10. Ultimately problems will be either in the product as a feature / bug, database, configurations or the ecosystem or similar logical areas. Isolating these early definitely helps.

11. Have you checked Liferay Learn, Liferay Blogs, Liferay Forums and Liferay Help Center plus Customer portal for similar problems / articles?

12. Are you at the latest minor patch version for your major release?

13. Are your customizations confirming to Liferay & Java official processes?

14. Are you compliant to Liferay support matrix?

Having a holistic debugging view like above can help support cut through lot of unnecessary iterations.

Keywords from Day 5 of Online Workshop on Development and Deployment of AIoT based solution for Industrial Applications by NSUT, Delhi

Further to Day 1 & 2 / Day 3 & 4 keywords posted in the past as per the links here, below are the keywords for Day 5: Keywords from Day 1 & 2 of Online Workshop on Development and Deployment of AIoT based solution for Industrial Applications by NSUT, Delhi | LinkedIn, Keywords from Day 3 & 4 of Online Workshop on Development and Deployment of AIoT based solution for Industrial Applications by NSUT, Delhi | LinkedIn:

Session 1:

Speakers:

Dr. Sudeepta Mishra | Department of Computer Science & Engineering (iitrpr.ac.in)

Sudeepta Mishra | LinkedIn

Raushan Kumar Singh | LinkedIn

Keywords:

TinyML 

MicroControllers and it’s advantages in IoT / AIoT

Hardware Software Co-design 

Hacking / confusing signals to IoT / Electromagnetic Pulses  

Internet of Battlefield Things (IoBT) 

Sensor hacking 

Use case for border fencing security 

Hand gesture recognition practical 

www.EdgeImpulse.com – Build datasets, train models, and optimize libraries to run directly on device; from the smallest microcontrollers to gateways with the latest neural accelerators (and anything in between).

Programming on TinyML

Compressing models 

Sin wave generation and detection on TinyML 

On-Device Training with TensorFlow Lite

TensorFlow Lite for Microcontrollers 

TensorFlow Lite 

Hex file for microcontroller – header file 

EloquentTinyML library 

Arduino

Raspberry Pi

Wokwi – Online ESP32, STM32, Arduino Simulator 

Training your model and putting it directly in firmware

Quantization in TinyML

Session 2:

Speakers:

Prof. SRN Reddy – DIC-IGDTUW 

CSE (igdtuw.ac.in) 

Keywords:

Practical approach to Smart IoT devices 

ARM processor 

Peripherals

Embedded and IoT devices differences

Framework of Smart IoT – Smartphone

Jetson Nano NVidia

Hardware components for smart IoT / SmartPhone

My_OS

Commercialization of Smart IoT

Smart Healthcare product

Aquaculture product

Challenges with Aquaculture product – Scaling, reactions with water of sensors & dirty solar panels

Environmental monitoring system

Various MTech / PhD thesis discussion & demos for prototypes & products

ETI Labs Pvt. Ltd.

Various boards for Arduino, 8051 and products / kits on top of it

Instrumentation for above

Industry 4.0 testbed

Button marking & stitching machine – autonomous system

AIoT in agriculture – Temperature, Humidity, CO2 & AQI monitoring

Drone with multi-spectral imaging for farming for NPK (Nitrogen, phosphorus, and potassium) monitoring with AI

Production monitoring with AIoT

Keywords from Day 3 & 4 of Online Workshop on Development and Deployment of AIoT based solution for Industrial Applications by NSUT, Delhi

Further to Day 1 & 2 keywords given here: Keywords from Day 1 & 2 of Online Workshop on Development and Deployment of AIoT based solution for Industrial Applications by NSUT, Delhi | LinkedIn, here are the keywords and details for Day 3 & 4 below:

Day 3:

Speakers:

Swagatam Das | LinkedIn

Faizanuddin Ansari | LinkedIn

Yogita – National Institute of Technology, Kurukshetra (nitkkr.ac.in)

Topics:

Data analysis from IoT – when to use traditional statistics from sensors instead of ML/AI

Data in motion – Streaming AIoT, how to handle it and related techniques like clustering, windows and so on

How to generate your own GAN

Parts of GANs

Basics of GAN

CIFAR10 dataset

Diffusion model

GAN code

Deep Learning – GAN / Diffusion / CNN

Explainability & interpretability in AI/ML and libraries associated with it

Day 4:

Speakers:

Dr. Dinesh K. Vishwakarma | LinkedIn

Manu Narula | LinkedIn

VINOD P | LinkedIn

Topics:

Smart Farming use case with IoT and AI – Types of sensors, uses, etc.

Detecting diseases from leaf pictures in smart farming

Distributed, Federated and Active learning in IoT plus techniques related to it like quantization

Machine Unlearning

Types of Machine Unlearning

Tiny LLM

Knowledge distillation

Machine Unlearning in LLM

Natural language processing in cyber security in Cyber Threat Intelligence

Definition of Cyber Threat Intelligence

Diamond model in Cyber Security

MITRE

Vulnerability databases

Named entity recognition, tagging and annotations

Web scraping

Building a cyber security dictionary

Natural language graphs and algorithms to process entities

STIX 2 (OASIS-OPEN.ORG)

Self-attention and semantic embedding

BERT

Fine tune BERT

VX Underground

Keywords from Day 1 & 2 of Online Workshop on Development and Deployment of AIoT based solution for Industrial Applications by NSUT, Delhi

Attending online “Five Days SERB sponsored Workshop on Development and Deployment of AIoT based solution for Industrial Applications” from 17th-21st June 2024

Organized by:

Dept. of Computer Science & Engineering

Netaji Subhas University of Technology, Delhi

110078, India

www.nsut.ac.in

Following Topics will be covered during the program.

• Introduction to AIoT, Sensors, AIoT Use Cases.

• Artificial Intelligence: A requirement for IoT

• Deployment for Raspberry PI

• Machine Learning for IoT, ML on IoT Data

• Deep Learning: A Review, Transfer learning for AIoT

• Deployment of ML/DL models over Web

• Federated Learning based Air pollution Monitoring

• IoT Security: Countermeasures

• Deploying DL model on Edge Devices

• An Assistive Solution for Visually Impaired

• Comparison of Edge Devices for AIoT Deployment

• Research Possibilities and Future Directions

https://drive.google.com/file/d/1UjWlSlhZJ-xZxAe5wtEAfVl-OCYrOOPK/view – Poster link

Keywords:

Day 1:

  • Grounding in LLMs
  • Fashion industry use case for AI
  • Licensed vs. unlicensed spectrum in India
  • https://edgeimpulse.com/ – LoRaWAN, LoRa – AIoT example
  • Elaboration of architecture for AIoT protocol LoRa and LoRaWAN deployed at one of the IITs plus fashion industry usecase

Day 2:

AIoT network simulation:

Automatic speech recognition:

  • Phonemes vs Morphemes
  • Disadvantages of traditional cascade models – errors are amplified
  • MFCC
  • LSTM
  • End to end model
  • Hugging face
  • Whisper model

Speakers:

What is OSGI? – As per Gemini & ChatGPT.

OSGi (Open Service Gateway Initiative) is a fundamental technology used in Liferay DXP (Digital Experience Platform). It provides a modular framework that allows Liferay to be built from reusable components called bundles. These bundles can be dynamically installed, started, stopped, updated, and uninstalled without restarting the entire platform.

Here’s how OSGi is used in Liferay:

  1. Modularity: OSGi enables Liferay’s modular architecture, where each functionality is encapsulated within a bundle. This modularity makes Liferay more flexible, scalable, and easier to maintain.
  2. Service Registry: Bundles can provide and consume services through OSGi’s service registry. This allows for loose coupling between bundles, making it easier to add, remove, or update components without affecting the entire system.
  3. Dynamic Management: OSGi allows for dynamic management of bundles. This means that bundles can be installed, started, stopped, updated, or uninstalled while Liferay is running, minimizing downtime and enabling hot deployment of new features or bug fixes.
  4. Versioning: OSGi supports versioning of bundles and packages, ensuring compatibility between different components and allowing for seamless updates without breaking existing functionality.
  5. Dependency Management: OSGi automatically manages dependencies between bundles, making it easier to resolve conflicts and ensuring that all required components are available.

Benefits of using OSGi in Liferay:

  • Improved Flexibility: The modular structure allows for easy customization and extension of Liferay’s functionality.
  • Increased Scalability: Bundles can be added or removed as needed, making it easier to scale the platform to meet changing requirements.
  • Simplified Maintenance: Modularization makes it easier to identify and fix bugs or issues, as well as to update specific components without affecting the entire system.
  • Enhanced Stability: The loose coupling between bundles makes Liferay more resilient to failures, as a single bundle’s failure is less likely to bring down the entire system.
  • Faster Development: The modular architecture enables parallel development of different components, speeding up the overall development process.

Liferay leverages OSGi to its fullest potential, providing a robust and flexible platform for building digital experiences. Whether you’re developing custom modules or extending existing functionality, OSGi makes it easier to manage and maintain your Liferay implementation.

Let me know if you have any other questions.

OSGi (Open Services Gateway initiative) is a Java framework for developing and deploying modular software programs and libraries. Liferay, a popular enterprise portal platform, leverages OSGi to provide a highly modular, flexible, and dynamic development environment. Here’s a breakdown of how OSGi is used in Liferay:

### 1. Modularity

OSGi allows Liferay to break down its large codebase into smaller, reusable modules. Each module (or bundle) can be independently developed, tested, and deployed. This modularity helps in managing the complexity of large applications and promotes code reuse.

### 2. Dynamic Updates

With OSGi, Liferay can update, add, or remove modules without requiring a full restart of the application server. This dynamic behavior minimizes downtime and enhances the agility of the development and deployment processes.

### 3. Service Layer

OSGi defines a service registry where modules can publish and consume services. In Liferay, this enables a decoupled architecture where different modules can interact through well-defined service contracts, improving maintainability and scalability.

### 4. Versioning

OSGi provides a robust versioning system that allows multiple versions of the same module to coexist. This capability is particularly useful in Liferay for managing dependencies and ensuring backward compatibility.

### 5. Dependency Management

OSGi handles dependencies between modules automatically. Liferay uses this feature to ensure that all required modules are available and correctly configured, reducing runtime errors and simplifying the deployment process.

### 6. Isolation

OSGi offers a high degree of isolation between modules, which helps in avoiding classpath issues and conflicts. In Liferay, this isolation ensures that changes in one module do not inadvertently affect others, enhancing the stability of the system.

### Implementation in Liferay

In Liferay, the OSGi framework is integrated as the core runtime environment. Here’s how Liferay implements OSGi:

– Liferay Modules: Liferay plugins, such as portlets, hooks, and themes, are developed as OSGi modules. These modules follow the OSGi lifecycle and can be managed through the Liferay Control Panel.

– Service Builder: Liferay’s Service Builder tool generates OSGi-compliant services, making it easier to develop and deploy service-based applications.

– Module Framework: Liferay provides a Module Framework that extends OSGi capabilities, offering additional features like module dependency management, lifecycle management, and configuration management.

### Benefits of Using OSGi in Liferay

– Improved Modularity: Simplifies development and maintenance by promoting a modular architecture.

– Enhanced Flexibility: Allows dynamic updates and reconfigurations without downtime.

– Better Resource Management: Efficiently handles dependencies and resource allocation.

– Scalability: Facilitates scalable application design through decoupled modules.

– Maintainability: Eases the process of updating and maintaining the application by isolating changes to specific modules.

In summary, OSGi plays a crucial role in making Liferay a more modular, dynamic, and robust platform for developing enterprise-grade applications.

Search related techno-functional tips – Part 1

  • We can search a lot of things using Elasticsearch / SOLR or similar appliances by Google etc. but we can’t load everything in memory generally. It will simply overwhelm the heap and the processing engine with lot of calculations depending on type of system.
  • That necessitates us to device proper UX / UI with below points:
  • Filter by timeline sliders.
  • Use lazy loading from backend where possible.
  • Restrict background results so that front end system is not overwhelmed.
  • Give advanced filtering capabilities by date, time, topic, NLP / semantic search, AI prompting and so on.
  • Be careful of ASC / DESC sorting and deep pagination.
  • Focus on boosting / accuracy / indexing / relevance / shards / full text search / caching / top queries / optimizations / categorization-based search / metrics measurement / relevance / privacy / security and so on.
  • What is important is to search everything but not return everything.
  • Search should be as specific as possible.
  • If you want all results in a return result, then search is not the optimal solution, better work with caching / DB / in-memory products / etc.
  • Website: www.HarwaniSystems.in
  • Blog: www.TechAndTrain.com/blog
  • LinkedIn: Neil Harwani | LinkedIn
  • Email: Neil@HarwaniSystems.in

How to debug Liferay? – Some pointers – Part 2

Practical tips to debug Liferay – further to my two earlier posts:

How to debug Liferay when nothing is printed in logs and there is a problem? – ChatGPT & BARD Generated | LinkedIn

How to debug Liferay? – Some pointers – Part 1 | LinkedIn

  • Setup automated heap dumps when out of memory happens.
  • Setup automated thread dumps when Liferay slows down.
  • Set up Glowroot persistence.
  • Set up Glowroot in central pattern in production especially.
  • Use ycrash.io to analyze thread and heap dumps if possible or an offline tool as per your policy.
  • Cannot emphasize enough on the above five.
  • Check Liferay logs.
  • Check DB connection pool settings via Hikari pool or others as set and it’s exhaustion.
  • Check configuration files: setenv.sh/bat server.xml, osgi/configs, other app server configurations, etc.
  • Check logs of Elasticsearch, Database, Webserver, Load balancer, Web application firewall, Content Delivery Network and more.
  • Use CAT API of Elasticsearch to check status of Elasticsearch.
  • Check if any schedulers or integrations are putting load on the system.
  • Check if things like caching, content delivery, security, etc. are as much as possible offloaded from app server.
  • Check network delays & errors across all component deployments and debug layer by layer for: LR, app nodes, WS, ES, DB, WAF, CDN and so on.
  • Automate via scripting monitoring of CPU, threads and Heap especially on all servers in deployment.
  • Analyze trends for traffic.
  • Website: www.HarwaniSystems.in
  • Blog: www.TechAndTrain.com/blog
  • LinkedIn: Neil Harwani | LinkedIn
  • Email: Neil@HarwaniSystems.in

Ideas on Innovation around Technology. We Thrive On Ideas. We are Learner Centered, Open Source & Digital Focused.