Documentation
¶
Overview ¶
Package extensions implements various helper addons for Colly
Index ¶
- func RandomMobileUserAgent(c *colly.Collector)
- func RandomUserAgent(c *colly.Collector)
- func Referer(c *colly.Collector)
- func URLLengthFilter(c *colly.Collector, URLLengthLimit int)
- type CDPDriver
- func (c *CDPDriver) Cache(request *http.Request, bodySize int, checkHeadersFunc colly.CheckHeadersFunc, ...) (*colly.Response, error)
- func (c *CDPDriver) CheckRedirect(f func(req *http.Request, via []*http.Request) error)
- func (c *CDPDriver) Cookies(url *url.URL) []*http.Cookie
- func (c *CDPDriver) Do(request *http.Request, bodySize int, checkHeadersFunc colly.CheckHeadersFunc) (*colly.Response, error)
- func (c *CDPDriver) GetJar() http.CookieJar
- func (c *CDPDriver) GetMatchingRule(domain string) *colly.LimitRule
- func (c *CDPDriver) GetTimeout() time.Duration
- func (c *CDPDriver) Init(jar http.CookieJar)
- func (c *CDPDriver) Jar(j http.CookieJar)
- func (c *CDPDriver) Limit(rule *colly.LimitRule) error
- func (c *CDPDriver) Limits(rules []*colly.LimitRule) error
- func (c *CDPDriver) Proxy(pf colly.ProxyFunc)
- func (c *CDPDriver) SetClient(client *http.Client)
- func (c *CDPDriver) SetCookies(url *url.URL, cookies []*http.Cookie) error
- func (c *CDPDriver) Timeout(t time.Duration)
- func (c *CDPDriver) Transport(t http.RoundTripper)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func RandomMobileUserAgent ¶
func RandomMobileUserAgent(c *colly.Collector)
RandomMobileUserAgent generates a random MOBILE browser user-agent on every requests
func RandomUserAgent ¶
func RandomUserAgent(c *colly.Collector)
RandomUserAgent generates a random DESKTOP browser user-agent on every requests
func Referer ¶
func Referer(c *colly.Collector)
Referer sets valid Referer HTTP header to requests. Warning: this extension works only if you use Request.Visit from callbacks instead of Collector.Visit.
func URLLengthFilter ¶
func URLLengthFilter(c *colly.Collector, URLLengthLimit int)
URLLengthFilter filters out requests with URLs longer than URLLengthLimit
Types ¶
type CDPDriver ¶
type CDPDriver struct {
// contains filtered or unexported fields
}
func ChromeDriver ¶
func ChromeDriver() *CDPDriver
func (*CDPDriver) CheckRedirect ¶
func (*CDPDriver) GetMatchingRule ¶
func (*CDPDriver) GetTimeout ¶
func (*CDPDriver) SetCookies ¶
func (*CDPDriver) Transport ¶
func (c *CDPDriver) Transport(t http.RoundTripper)
Source Files
Click to show internal directories.
Click to hide internal directories.