Robert 雜記

reactor AsyncioSelectorReactor issue

Exception: The installed reactor (twisted.internet.selectreactor.SelectReactor) does not match the requested one (twisted.internet.asyncioreactor.AsyncioSelectorReactor)

command

install

1

pip install flask

command

install

1

pip install scrapyrt

說明

check scrapy methoud

• JavaScript Require?
• check API supported - Network > Fetch/XHR
• check pagination

computerdeals

get products

create spider

1
2
3

(myenv10_scrapy) D:\work\run\python_crawler\107-selenium\silkdeals>scrapy genspider computerdeals slickdeals.net/computer-deals
Created spider 'computerdeals' using template 'basic' in module:
  silkdeals.spiders.computerdeals

Command

install

install selenium

1

pip install selenium

install webdriver_manager(support auto download driver)

1

pip install webdriver_manager

install scrapy-selenium

1

pip install scrapy-selenium

livecoin

run by Chrome

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

-- https://web.archive.org/web/20200116052415/https://www.livecoin.net/en
function main(splash, args)
  -- splash private memoy(enable by default)
  -- if not enable no see RUR data
  -- ************* seem not work ************ 
  splash.private_mode_enabled = false
  
  url = args.url
  assert(splash:go(url))
  assert(splash:wait(1))
  rur_tab = assert(splash:select_all(".filterPanelItem___2z5Gb "))
  -- index start from 1
  rur_tab[5]:mouse_click()
  assert(splash:wait(5))
  
  splash:set_viewport_full()
  return splash:png()
end

說明

Browser Engine

• V8 Engine : Chrome
• Spider Monkey : Firefox
• Apple WebKit : Safari
• Chakra : Microsoft Edge, Splash

說明

5 components

• Spiders
• Spiders (spider Middleware-extracting data)
  • scrapy.spider
  • crawlspider
• Pipelines
• Middleware(Downloader Middeware)
• Engine
• Scheduler

scrapy install

install

python 3.11 有問題, python 3.10 ok

1
2
3
4
5
6
7
8

# install myenv10_scrapy
rem cd \app\python_env\
rem py -3.10 -m virtualenv myenv10_scrapy
# install 
pip install scrapy 
pip install pylint 
pip install autopep8
pip install ipython