System Design 1 - MapReduce

I was always wondered the difference between clusters and horizontal scaling. I was under an impression that both are same. Recently i was reading this link about the Map reduce that forced me to think again on the difference between clustering and horizontal scaling. I have discussed the same with one of my peer as well to get the clarity. This is what my understanding regarding the same.

Horizontal scaling is used in the case of scalability. If your application getting more traffic than usual then you need to scale the system to handle more requests like you add more suppliers in the restaurants when there are more people visiting the same. So each supplier can independently work to attain his goal. In our case we will add more hosts to the application which will handle requests completely independent. But who will tell these requests to go to this particular host ? load balancer comes into the picture now. Load balancer will route the request to hosts which having lesser load at that point of time. Load balancer is the external facing server , so it has to be highly available. So we must need to keep a backup node for the load balancer. So the time when the main load balancer dies then the backup should take the control.

Clustering is used when we have a computationally large problem and we need more resources to solve it parallel. So we will break the problem into subtasks and assign different tasks to different nodes and the results will be combined together to make the final result. This is distributed computing. MapReduce is helping us to create these distributed tasks.

Map will break down the tasks and give to all the mapping node, there will be a aggregator which will aggregate and give the result to the reducer. Reducer will reduce the map results to the final result.

You can read more about the map reduce function in this article.
Want to understand mapreduce through an example, this article is a good start.
  

Building a Spring boot application in Ubuntu 16.04 using gradle and Running a headless chrome from selenium

Hello everyone, this is my side by side blog on how to build my spring boot application in Ubuntu 16.04. I have already a project with basic spring boot specification which has build using gradle. This is my learning project on gradle and spring boot. So we all need a unix based os for deploying these to a cloud based hosting like digital ocean. So lets start.

These are the steps that i have already done.
1. Installed gradle in ubuntu 16.04 by following this article.
2. Copy the spring boot project to ubuntu box.
3. Installed Mongodb on Ubuntu machine by following this article
4. Installed Java 1.8

Oh.. Gradle !! Yeah it is a build automation tool that will take care of all the dependency and build tasks that we need to do while creating a project. Of course you can use other build tools like Maven,ant etc. But gradle comes with more power that other build tools in java.

Heard about gradlew ? It is worth looking into this if you are using gradle which will reduce your headache of installing gradle and will escape you from the version mismatch drama. Gradle vs gradlew ? . Gradle wrapper command will create gradlew.

Ok. lets come back to our Ubuntu Box. Our objective is to run spring boot in Ubuntu machine and the boot application is having a dependency on mongodb services. So we need to setup that as well.

You just need to go to the root directory where the build.gradle file exist and execute the command gradle bootRun. This will launch the spring boot process and you will something like below.

Yay !! spring boot has started !!.

One of my requirement for this spring boot project is that i want to run a selenium scrapper from this project that too headlessly using chrome. This link provides you more information on how can you install headless selenium.

Luckily Chrome has rolled out chrome-59 with headless feature which we can use for our experiment. Get the latest version of the chrome from this link.

So  i have installed google-chrome 59 and all ready for running my headless chrome instance with selenium with spring boot. I have tested my application code in my windows machine and it worked like a charm. Now i am testing it on my Ubuntu server. fingers crossed !

It throws me unable to start exception. Mostly its because Ubuntu server is not able to start it on a graphical way. Lets investigate.

woo.. after a long 3-4 hours i could run the chrome 59 headless from selenium. Initially i was getting the same above error whatever i changed. I did a lot of googling to find out why the same exception coming. It was due to one of my big blunder of adding the wrong system property. I was adding the actual chrome installation path in the webdriver.chrome.driver property. Everything started working once i replaced the same with chrome-driver location.

For that i have downloaded the chrome-driver from the Chromium.org download site. And i did something similar including the symlink like this.

This particular script written in Github shows all the steps that we need to do to install the Chrome+selenium combo in your Unix boxes.

So here is my complete java code that running the selenium which invoke the headless google-chrome in Ubuntu machine.

And we need to pass the extra arguments for running the chrome in headless mode(Note that this feature available only google-chrome-59 onwards). The versions of the applications that used as follows.

Java - 1.8
Selenium - 3.*
google-chrome - 59
chromeDriver - 2.3

That's it guys. We have successfully ran our spring boot application which can invoke chrome instance headless way from selenium. Tune in for more. Thanks a lot.  

Pointers for Spring boot + Mongodb applications

Connecting the mongodb database to spring boot is very important and powerful. The most easy way to persist your application data is to store it somewhere. We all do it by storing it into some kind of databases. In these types of databases i found out that mongodb is very easy to handle and configure as it is schema-less. So i dont need to create any sort of database design instead i can send my json file for saving it and i can retrieve it back.

First of all, lets look into some basic links that needed.

1a. Install mongodb on ubuntu 16.04 - Link
1b. Install mongodb on windows 7 -  Link

2. You can use mongo command to see the CLI of mongo instance that is running, a few basic         commands as follows.
    - show dbs
    - show collections
    - db.{collection_name}.find()

If you want to understand how MongoDB can be used in spring boot then this link will give you a good idea to do it.  In this article you can see that there are two types of connection that can be used to connect to mongodb and i found out that mongotemplate is having much more flexibility.

More complex mongoTemplate queries can be made by following this article.


============================== References==================================
It is a good starting point for starting spring boot applications. It follows very good design patterns to follow. Check here

You might also need to get some of the annotations that are needs to be used. check this

NLP - emotion analysis - links

Very important links

Python-NRC-example - Provide basic example on how to use NRC emotion lexicon

NRC-Emotion-Lexicon - Download the lexicon by Saif

Datasets for emotion analysis

Enron emotion - Classified enron dataset for emotion analysis

SentiSense lexicon -  This is a different lexicon for affective words(*)

Emotion analysis examples

Beefoo Github - Provide implementation of emotion analysis

Kaggle Trump-Hillary -  A basic use case of emotion based on tweets

Trump analysis in R  - Worth checking out this one if you have knowledge in R

Hillary emails in R - Again good data analysis on emails but in R

Tweet Emotion in R - Tweet an

Some Deep Learning

go deep

Understanding Node+Express application flow.

As installation has explained in the last article now we want to start with the project and structure. It is very important to understand the proper directory structure of the node app since we need to map the relevant files in the right directory to align with the MVC pattern that node follows. I have found out the following article which explains the node directory structure perfectly.

Check this article

Confused over export and require ? 

I was confused too much on export, module and require stuff of nodejs until i have came across the following article which deals with the same. what i have taken from the article is that, a file is considered as a module in nodejs and if you want to use it in another file(module) then you need to export it. Simple stuff !!

Check this article

Router everywhere !!

If you are start making Restful webservices we need to use routers very frequently to create different routes for our RestWebservices. They have introduced mini express in router and if you want to understand how the router works and the different functions that can be done in router like params,use,get etc.

Check this article for more information

Understanding the basic components using in the app.js

It is essential to know the components that we are using in our landing file that is app.js and their role in this particular context. So i just thought of going through each package that we are using in the app.js file.

Path - Just to know 

Path is the package  to deal with all the directory level operations to be done. __dirname gives you the current working directory and you can do many other operations like join(), basename() etc.

Check this link for more details 

Error handling in Node

Error handling is very important feature every application should have. It will make the application maintainable and debuggable. The Nodejs style of error handling is little different which involves middle-wares which will make your job damn easy. check the links below.

Link 1

Link 2

Logging

Logging can be one of the important thing is to be done with any software application. Winston is one of the finest logger that you can use for Nodejs. Read more 

All about passing the data in Nodejs routes

Passing the data in the routes are very important to deal with the flexibility of the microservcie framework. As Node stands out as an exceptional framework to create microservices it is important to know how to pass the data.

This article talks about each and everything about passing the data between routes. 

Nodejs installation and basics

Nodejs is getting popularity as Javascript can be used as a complete stack and JS is  the only language that you need to know before creating a web application. Yeah nothing else.

My requirements are like to install the nodejs in a unix based server, preferably centos or ubuntu. simple commands will help you to install nodejs. The following link will help you to install node perfectly to your centos system.

Please check this link 

One of the issue that i faced just after installing the nodejs was my command prompt was not understanding the node or npm as commands. What i understood is that the installation was installing node and npm to /usr/local/bin/ location. But the command line understand the binaries in the /usr/bin location. So we need to map this using synlink.

ln -s /usr/local/bin/node /usr/bin/node
ln -s /usr/local/lib/node /usr/lib/node
ln -s /usr/local/bin/npm /usr/bin/npm
ln -s /usr/local/bin/node-waf /usr/bin/node-waf

This link will help you to understand the what is going on.

Now install express with npm and the same synlinking we need to do for each package that we create.

 ln -s /usr/local/bin/express /usr/bin/express
 ln -s /usr/local/bin/node-supervisor /usr/bin/node-supervisor
 ln -s /usr/local/bin/supervisor /usr/bin/supervisor
 ln -s /usr/local/bin/n /usr/bin/n

It will be good to go through this link before we start installing new packages. This will help you understand how to install a npm package.

 n stable

it will be good if we install any daemon based package to manage our nodejs app. So it will take care of handling the nodejs instance running.

 npm install forever
 npm install forever-monitor
 ln -s /usr/local/bin/forever /usr/bin/forever

Now you can start nodejs daemon in forever as -
   forever start /tmp/node/UpgradeChecker/app/bin/www


If you want to make this as a custom service then check this link. If you do then you can start your app like,
 service nodeapp start

Python text to speach

It is a good comparison of various tools available for text to speech :

http://fossforce.com/2015/04/an-in-depth-look-at-text-to-speech-in-linux/

Setting the mbrola -- http://askubuntu.com/questions/554747/how-to-install-more-voices-to-espeak

Worth checking mary - https://sites.google.com/site/guenterbartsch/blog/maryttspythonandpulseaudio

Example python code -http://stackoverflow.com/questions/6772710/function-doesnt-work-the-second-time-its-called-in-a-thread-pyttsx-module

linux processes - https://idea.popcount.org/2012-12-11-linux-process-states/

Looking for natural sounds ? -http://askubuntu.com/questions/21811/how-can-i-install-and-use-text-to-speech-software