Documentation
¶
Index ¶
- type CantabularClient
- type DatasetAPIClient
- type Error
- type FilterAPIClient
- type FilterInfo
- type Generator
- type InstanceComplete
- func (h *InstanceComplete) GetS3ContentLength(_ context.Context, _ *event.ExportStart, isPublished bool, filename string) (int, error)
- func (h *InstanceComplete) Handle(ctx context.Context, _ int, msg kafka.Message) error
- func (h *InstanceComplete) ProduceExportCompleteEvent(ctx context.Context, e *event.ExportStart, rowCount int32, fileName string) error
- func (h *InstanceComplete) UpdateFilterOutput(ctx context.Context, e *event.ExportStart, size int, isPublished bool, ...) error
- func (h *InstanceComplete) UpdateInstance(ctx context.Context, e *event.ExportStart, size int, isPublished bool, ...) error
- func (h *InstanceComplete) UploadCSVFile(ctx context.Context, e *event.ExportStart, ...) (string, int32, string, error)
- func (h *InstanceComplete) ValidateInstance(i dataset.Instance) (bool, error)
- type S3Client
- type VaultClient
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type CantabularClient ¶
type CantabularClient interface {
StaticDatasetQueryStreamCSV(ctx context.Context, req cantabular.StaticDatasetQueryRequest, consume cantabular.Consumer) (rowCount int32, err error)
}
CantabularClient contains the required method for the Cantabular Client
type DatasetAPIClient ¶
type DatasetAPIClient interface {
GetInstance(ctx context.Context, userAuthToken, serviceAuthToken, collectionID, instanceID, ifMatch string) (i dataset.Instance, eTag string, err error)
PutVersion(ctx context.Context, userAuthToken, serviceAuthToken, collectionID, datasetID, edition, version string, v dataset.Version) error
}
DatasetAPIClient contains the required method for the Dataset API Client
type Error ¶
type Error struct {
// contains filtered or unexported fields
}
Error is the handler package's error type. Is not meant to be compared as a type, but information should be extracted via the interfaces it implements with callback functions. Is not guaranteed to remain exported so shouldn't be treated as such.
type FilterAPIClient ¶ added in v1.0.0
type FilterAPIClient interface {
GetDimensions(ctx context.Context, userAuthToken, serviceAuthToken, collectionID, filterID string, q *filter.QueryParams) (dims filter.Dimensions, eTag string, err error)
GetOutput(ctx context.Context, userAuthToken, serviceAuthToken, downloadServiceToken, collectionID, filterOutput string) (m filter.Model, err error)
UpdateFilterOutput(ctx context.Context, userAuthToken, serviceAuthToken, downloadServiceToken, filterOutputID string, m *filter.Model) error
}
type FilterInfo ¶ added in v1.13.0
type Generator ¶ added in v0.4.0
Generator contains methods for dynamically required strings and tokens e.g. UUIDs, PSKs.
type InstanceComplete ¶
type InstanceComplete struct {
// contains filtered or unexported fields
}
InstanceComplete is the handle for the InstanceCompleteHandler event
func NewInstanceComplete ¶
func NewInstanceComplete(cfg config.Config, c CantabularClient, d DatasetAPIClient, f FilterAPIClient, sPrivate, sPublic S3Client, v VaultClient, p kafka.IProducer, g Generator) *InstanceComplete
NewInstanceComplete creates a new InstanceCompleteHandler
func (*InstanceComplete) GetS3ContentLength ¶ added in v0.6.0
func (h *InstanceComplete) GetS3ContentLength(_ context.Context, _ *event.ExportStart, isPublished bool, filename string) (int, error)
GetS3ContentLength obtains an S3 file size (in number of bytes) by calling Head Object
func (*InstanceComplete) ProduceExportCompleteEvent ¶
func (h *InstanceComplete) ProduceExportCompleteEvent(ctx context.Context, e *event.ExportStart, rowCount int32, fileName string) error
ProduceExportCompleteEvent sends the final kafka message signifying the export complete
func (*InstanceComplete) UpdateFilterOutput ¶ added in v1.0.0
func (h *InstanceComplete) UpdateFilterOutput(ctx context.Context, e *event.ExportStart, size int, isPublished bool, s3Url, filename string) error
func (*InstanceComplete) UpdateInstance ¶
func (h *InstanceComplete) UpdateInstance(ctx context.Context, e *event.ExportStart, size int, isPublished bool, s3Url, filename string) error
UpdateInstance updates the instance downlad CSV link using dataset API PUT /instances/{id} endpoint if the instance is published, then the s3Url will be set as public link and the instance state will be set to published otherwise, a private url will be generated and the state will not be changed
func (*InstanceComplete) UploadCSVFile ¶
func (h *InstanceComplete) UploadCSVFile(ctx context.Context, e *event.ExportStart, req cantabular.StaticDatasetQueryRequest, isPublished, isCustom bool) (string, int32, string, error)
UploadCSVFile queries a static dataset to cantabular using the provided request, transforms the response to a CSV format and streams the data to S3. If the data is published, the S3 file will be stored in the public bucket If the data is private, the S3 file will be stored in the private bucket (encrypted or un-encrypted depending on EncryptionDisabled flag) Returns S3 file URL, file size bytes, number of rows, and any error that happens.
func (*InstanceComplete) ValidateInstance ¶ added in v0.3.0
func (h *InstanceComplete) ValidateInstance(i dataset.Instance) (bool, error)
ValidateInstance validates the instance returned from dp-dataset-api Returns isPublished bool value and any validation error
type S3Client ¶
type S3Client interface {
Head(key string) (*s3.HeadObjectOutput, error)
UploadWithContext(ctx context.Context, input *s3manager.UploadInput, options ...func(*s3manager.Uploader)) (*s3manager.UploadOutput, error)
UploadWithPSK(input *s3manager.UploadInput, psk []byte) (*s3manager.UploadOutput, error)
BucketName() string
}
S3Client contains the required method for the S3 Client
type VaultClient ¶ added in v0.3.0
VaultClient contains the required methods for the Vault Client
Source Files
Directories