Need Proxy?

BotProxy: Rotating Proxies Made for professionals. Really fast connection. Built-in IP rotation. Fresh IPs every day.

Find out more

twisted critical unhandled error on scrapy tutorial

Question

I'm new in programming and I'm trying to learn scrapy, using scrapy tutorial: http://doc.scrapy.org/en/latest/intro/tutorial.html

So I ran "scrapy crawl dmoz" command and got this error:

2015-07-14 16:11:02 [scrapy] INFO: Scrapy 1.0.1 started (bot: tutorial)
2015-07-14 16:11:02 [scrapy] INFO: Optional features available: ssl, http11
2015-07-14 16:11:02 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE':     'tu
torial.spiders', 'SPIDER_MODULES': ['tutorial.spiders'], 'BOT_NAME':   'tutorial'}

2015-07-14 16:11:05 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsol
e, LogStats, CoreStats, SpiderState
Unhandled error in Deferred:
2015-07-14 16:11:06 [twisted] CRITICAL: Unhandled error in Deferred:
2015-07-14 16:11:07 [twisted] CRITICAL:

I'm using windows 7 and python 2.7. Anybody knows what's the problem? How could I fix that?

EDIT: My spider file code is:

# This package will contain the spiders of your Scrapy project
#
# Please refer to the documentation for information on how to create and manage
# your spiders.
import scrapy


class DmozSpider(scrapy.Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
     start_urls = [
    "http://www.dmoz.org/computers/programming/languages/python/books/",
    "http://www.dmoz.org/computer/programming/languages/python/resources/"
]

    def parse(self, response):
        filename = response.url.split("/")[-2] + '.html'
        with open(filename,'wb') as f:
            f.write(response.body)

items.py code:

import scrapy

class DmozItem(scrapy.Item):
    title = scrapy.Field()
    link = scrapy.Field()
    desc = scrapy.Field()

pip list:

  • bootstrap-admin (0.3.3)
  • cffi (1.1.2)
  • characteristic (14.3.0)
  • cryptography (0.9.3)
  • cssselect (0.9.1)
  • Django (1.7.7)
  • django-auth-ldap (1.2.4)
  • django-debug-toolbar (1.3.0)
  • django-mssql (1.6.2)
  • django-pyodbc (0.2.6)
  • django-pyodbc-azure (1.2.2)
  • django-redator (0.2.3)
  • django-reversion (1.8.5)
  • django-summernote (0.6.0)
  • django-windows-tools (0.1.1)
  • django-wysiwyg-redactor (0.4.3.2)
  • enum34 (1.0.4)
  • ez-setup (0.9)
  • flup (1.0.2)
  • idna (2.0)
  • ipaddress (1.0.13)
  • iso8601 (0.1.4)
  • logging (0.4.9.6)
  • lxml (3.4.4)
  • mechanize (0.2.5)
  • MySQL-python (1.2.4)
  • pbr (0.10.8)
  • Pillow (2.7.0)
  • pip (7.1.0)
  • pyasn1 (0.1.8)
  • pyasn1-modules (0.0.6)
  • pycparser (2.14)
  • pymongo (2.6)
  • pyodbc (3.0.7)
  • pyOpenSSL (0.15.1)
  • pypm (1.4.3)
  • python-ldap (2.4.18)
  • pythonselect (1.3)
  • pywin32 (218.3)
  • queuelib (1.2.2)
  • Scrapy (1.0.1)
  • selenium (2.44.0)
  • service-identity (14.0.0)
  • setuptools (18.0.1)
  • six (1.9.0)
  • sqlparse (0.1.15)
  • stevedore (1.3.0)
  • Twisted (15.2.1)
  • virtualenv (1.11.6)
  • virtualenv-clone (0.2.5)
  • virtualenvwrapper (4.3.2)
  • virtualenvwrapper-powershell (12.7.8)
  • w3lib (1.11.0)
  • xlrd (0.9.2)
  • zope.interface (4.1.2)

Thx for the attention and sry for my poor English, isn't my native language.

Answer

I'm beginning to learn scrapy as well and encounter the same question with yours. After struggling with it for an afternoon, finally I found it's due to the pywin32 module only download without install. You can try input the command below in the cmd to finish the pywin32 module install and try crawl again:

python python27\scripts\pywin32_postinstall.py -install

I hope it will help!

cc by-sa 3.0

See also: