especs_sitio_proceso.txt

Para el crawler de la seccion 'Hemeroteca' del sitio 'Proceso' se requirio lo siguiente (a parte de scrapy):
-docker
-splash
-scrapy-splash

=================================================
Instalacion DOCKER, ubuntu 16.04
=================================================
	$ sudo apt-get update
	$ sudo apt-get upgrade
	
	$ sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D

	$ sudo apt-add-repository 'deb https://apt.dockerproject.org/repo ubuntu-xenial main'
	$ sudo apt-get update
	$ sudo apt-get install docker-engine


> Inicializar/detener Docker:
	$ sudo service docker start/stop

> Version:
	$ docker --version


Consulta: https://thishosting.rocks/install-docker-on-ubuntu/





=================================================
Instalacion SPLASH
=================================================
	$ sudo docker pull scrapinghub/splash


> Inicializar el contenedor:
	$ sudo docker run -p 8050:8050 scrapinghub/splash     ## con esto splash esta disponible en puerto 
	                                                      ## 8050 (http) en navegador (localhost:8050)


Consulta: http://splash.readthedocs.io/en/latest/install.html





=================================================
Instalacion scrapy-splash
=================================================
	$ pip install scrapy-splash


> Configuraciones en settings.py:

	* Para usar scrapy-splash en un proyecto primero se necesita habilitar el middleware:

		DOWNLOADER_MIDDLEWARES = {
			'scrapy_splash.SplashCookiesMiddleware': 723,
			'scrapy_splash.SplashMiddleware': 725,
			'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
		}


	* Agregar el SPLASH_URL:

		SPLASH_URL = 'http://localhost:8050/'


	* Habilitar SplashDeduplicateArgsMiddleware, que permite ahorrar espacio en disco evitando almacenar multiples argumentos duplicados de Splash.

		SPIDER_MIDDLEWARES = {
			'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
		}


	* Configurar DUPEFILTER_CLASS y almacenamiento chache:

		DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
		HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'


Consulta: https://blog.scrapinghub.com/2015/03/02/handling-javascript-in-scrapy-with-splash/
					https://github.com/scrapy-plugins/scrapy-splash