Documentation
¶
Index ¶
- Constants
- func FailIfNeedLogin(body *string)
- func GetFilePathListWithPattern(basePath, fileNamePattern string) []string
- func GetPathWithCreation(subdirs string) (string, error)
- func GetPathWithCreationWithBase(base, subdirs string) (string, error)
- func LoadCookiesFile(filePath string) ([]*http.Cookie, error)
- func LoadCookiesFileToString(filePath string) (string, error)
- func MergeSpaces(str *string) string
- func NewColly() *colly.Collector
- func NewQueue() *queue.Queue
- func ReadEntireFile(fullPath string) string
- func StripAllSpaces(str *string) string
Constants ¶
View Source
const CollectorPathPrefix = "collector/"
View Source
const RequestInterval = 3 * time.Second
View Source
const RequestTimeout = 5 * time.Minute
Variables ¶
This section is empty.
Functions ¶
func FailIfNeedLogin ¶
func FailIfNeedLogin(body *string)
FailIfNeedLogin will fail the execution if the page requires login.
func GetFilePathListWithPattern ¶
GetFilePathListWithPattern returns the full paths for files matching the pattern in the base path.
func GetPathWithCreation ¶
GetPathWithCreation returns the concatenated path with output path and have them created in advance.
func GetPathWithCreationWithBase ¶
GetPathWithCreationWithBase returns the concatenated path with base path and have them created in advance.
func LoadCookiesFile ¶
LoadCookiesFile loads the external cookies file.
func LoadCookiesFileToString ¶
LoadCookiesFileToString loads the external cookies file.
func MergeSpaces ¶
MergeSpaces merges all space characters into one regular space.
func NewColly ¶
func NewColly() *colly.Collector
NewColly creates the Colly Collector based on my needs.
func ReadEntireFile ¶
ReadEntireFile reads the entire file to a string.
func StripAllSpaces ¶
StripAllSpaces removes all space characters.
Types ¶
This section is empty.
Source Files
Click to show internal directories.
Click to hide internal directories.