README
¶
Goscrapyd 
GoScrapyd is a web-based cluster management tool designed to simplify the management and monitoring of your Scrapyd instances. Built almost entirely in Go, it provides a modern, user-friendly interface with enhanced security and advanced features.
Features
- Light/Dark mode UI
- User accounts (privileged/unprivileged)
- Persisted settings (settings automatically applied to every task/spider run)
- Job lifecycle tracking (tracks which user started each job/task)
- Text search for tasks/jobs
- Native support for HTTPS via Let's Encrypt certificates
Run Locally
-
Install Go
-
Clone the repository
-
Run
go mod tidyfrom the cloned repository -
Build the executable by running
go build -o <desired_executable_name_here> .\cmd\web\ -
When running the program for the first time, use the command
<your_executable_here> -create-default-userto create a defaultadmin:adminuser account. This account can be used for first login. Once logged in you have the ability to easily create more of them and/or secure this one with a better password and username via the Web UI -
See Environment variables for other settings that should be configured
Environment Variables
The python-path environment variable is required, and it should point to a Python interpreter on your system.
-
The following environment variables are optional, but by default, they are insecure. It is strongly recommended that you change them:
cookie-secret-key- A secret key used for encrypting or signing cookies. Change this to a strong, random value for better security.session-secret-key- A secret key used to protect/secure user sessions.scrapyd-secret- If your Scrapyd nodes are authenticated, this secret is used to encrypt your Scrapyd credentials at rest in the database.
For all environment variables, please refer to main.go.
If you're interested in running an HTTPS server, please also check server.go in addition to main.go. Basic familiarity with Go is expected, as detailed instructions are not currently provided for this.
Directories