Commit 4e30de90 authored by Renán Sosa Guillen's avatar Renán Sosa Guillen

crawlers

parent 4be13ddf
No preview for this file type
No preview for this file type
No preview for this file type
......@@ -19,7 +19,7 @@ NEWSPIDER_MODULE = 'diarioCoLatino.spiders'
#USER_AGENT = 'diarioCoLatino (+http://www.yourdomain.com)'
# Obey robots.txt rules
ROBOTSTXT_OBEY = True
# ROBOTSTXT_OBEY = True
# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32
......
No preview for this file type
No preview for this file type
No preview for this file type
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment