crawler

package

v0.0.0-...-9831a62 Latest Latest Go to latest Published: Oct 19, 2021 License: GPL-3.0 Imports: 5 Imported by: 0

Details

This section is empty.

This section is empty.

This section is empty.

type ConcurrentCrawler struct {
	// contains filtered or unexported fields
}

ConcurrentCrawler limits number of concurrent requests.

func WithConcurrencyLimit(crawler crawler, limit int) *ConcurrentCrawler

WithConcurrencyLimit adds concurrency limit to the crawler.

func (c *ConcurrentCrawler) Crawl(ctx context.Context, u *url.URL) ([]byte, error)

Crawl downloads a page.

type Crawler struct {
	// contains filtered or unexported fields
}

Crawler crawls web and downloads stuff.

func New() *Crawler

New returns a new crawler.

func (c *Crawler) Crawl(ctx context.Context, url *url.URL) ([]byte, error)

Crawl downloads a page.