Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type HTTPStatusError ¶
type HTTPStatusError struct {
S string
}
HTTPStatusError ...
func (*HTTPStatusError) Error ¶
func (e *HTTPStatusError) Error() string
Error for the error interface
type HttpClient ¶
type HttpClient struct {
*generator.HTTPHeaderGenerator
// contains filtered or unexported fields
}
func NewHttpClient ¶
func NewHttpClient(awsServiceAddress string, config *ScraperConfig) *HttpClient
func (*HttpClient) Close ¶
func (h *HttpClient) Close()
func (*HttpClient) WithRetries ¶
func (h *HttpClient) WithRetries(requestRetryCount int, f func() error) error
type ScraperClient ¶
type ScraperClient interface {
WithRetries(times int, f func() error) error
Do(request *http.Request) (*http.Response, error)
}
ScraperClient is some implementation of a http client usable for scraping
type ScraperConfig ¶
type ScraperConfig struct {
ElasticAssignmentTimeout int
RequestTimeout int
RequestRetryCount int
ElasticIpRetryCount int
}
func GetScraperConfig ¶
func GetScraperConfig() *ScraperConfig
type SimpleScraperClient ¶
type SimpleScraperClient struct {
*generator.HTTPHeaderGenerator
// contains filtered or unexported fields
}
SimpleScraperClient handles retries and setting random headers for scraping
func NewSimpleScraperClient ¶
func NewSimpleScraperClient() *SimpleScraperClient
NewSimpleScraperClient returns an initialized SimpleScraperClient
func (*SimpleScraperClient) WithRetries ¶
func (s *SimpleScraperClient) WithRetries(times int, f func() error) error
WithRetries calls f with retries
Source Files
Click to show internal directories.
Click to hide internal directories.