Skip to content

crawlersNoticias
crawlersNoticias
Commit 18044725 authored by Mario Chirinos's avatar Mario Chirinos
Browse files
Options

dont crawl disabled

parent 555d79dd
Show whitespace changes
Inline Side-by-side
Showing with 2 additions and 0 deletions
scripts/crawlAll.py
View file @ 18044725
...@@ -26,6 +26,8 @@ def main(argv): ...@@ -26,6 +26,8 @@ def main(argv):
continue continue
with open(path+'settings.json') as json_file: with open(path+'settings.json') as json_file:
cfg = json.load(json_file) cfg = json.load(json_file)
if not cfg["enabled"]:
continue
logfile.write("Crawler "+cfg["crawler"]+" started at: " +datetime.datetime.now().strftime("%Y-%m-%d, %H:%M:%S")+"\n") logfile.write("Crawler "+cfg["crawler"]+" started at: " +datetime.datetime.now().strftime("%Y-%m-%d, %H:%M:%S")+"\n")
new_cwd = os.getcwd()+"/spiders/"+cfg["crawler"] new_cwd = os.getcwd()+"/spiders/"+cfg["crawler"]
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment