Scrapy - Get started !!

Scrapy is a python module for web scraping. If you want to scrape data in the websites in a automated way then this is one of the best option available in the techno-space. writing a python script with scrapy will create a crawler and it will scrape the data from the web sites that you intended to scrap.

Here i am listing few interesting links for setting and using the python scrapy !!

• This is one of the starting place for scrapy, which explains the setup and usage - Check here

• This is another wonderful article on scrapy !! Explains better - See here 

• Most probably you will get confused with xPath selection, If you want to look into it in detail then Check this and this

Use developer tools for xPath

Go to the inspect element of the required field in Chrome developer tool and right click+copy xpath

Right-click on the <tr> and copy the xpath and print it on the developer tool console to make sure that whether you got the correct xpath.

Here you can see that the xpath has selected the entire <tr>. now we can easily scrape the data with the scrapy's methods.

Run windows 7 in a virtualbox from Ubuntu 14.04

Step 1 : Open terminal and type,

sudo apt-get install virtualbox 

Virtualbox will get installed in 5 minutes.




Step 2 : Open virtualbox and create a new virtual machine as you want. Easy steps to be followed and will land in this state




Step 3 : Right click on the virtual machine listed in the left panel and select settings which will land you in the following page.

Click on storage

Click on Empty

Now you will see this page.

Step 4 : Click on the cd-image on the right panel near CD/DVD

Now choose a virtual CD/DVD file

Choose your iso file

Done !!

Now go to your virtualbox and select the vm and click RUN