Using an External Process in Your Nom Nom App

This article explains how to leverage an external process from your Nom Nom App. Specifically it discusses how to use the Scrapy python package. It is assumed that the reader is already familiar with the information in the Creating Your First Nom Nom App article.

Full Example App

First, install Scrapy. Then create a new Nom Nom app.

nnd engine-tools create-new scrapy-example

And change the contents of the pkg/executable.py to this code:

import logging
import scrapy

from nomnomdata.engine import Engine
from scrapy.crawler import CrawlerProcess

logger = logging.getLogger("engine.scrapy-example")

class TestSpider(scrapy.Spider):
    name = "test"
    allowed_domains = ["webscraper.io"]
    start_urls = ["https://webscraper.io/test-sites/tables"]

    def parse(self, response):
        logger.info(response.body)

engine = Engine(
    uuid="CHANGE-ME-PLEASE",
    alias="Scrapy Example",
    categories=["general"],
)

@engine.action(
    display_name="Run Scrapy Spider",
    description="",
)
def run_spider(parameters):
    process = CrawlerProcess()
    process.crawl(TestSpider)
    process.start()

The example above does not take in any input parameters.

CrawlerProcess

This utility class allows you to spawn the external process that your Nom Nom App will interact with. The process will still run inside the context of the Docker container where the rest of your code is running. The CrawlerProcess class has a method named crawl that takes an object based on scrapy.Spider that contains the details about the URL's that you want to examine. The class TestSpider represents this object in the sample code above.

An even more detailed example is available on the Scrapy website.