goal

Goal: Go augmented language

Goal is an augmented version of the Go language, which combines the best parts of Go, bash, and Python, to provide and integrated shell and numerical expression processing experience, which can be combined with the yaegi interpreter to provide an interactive "REPL" (read, evaluate, print loop).

Goal transpiles directly into Go, so it automatically leverages all the great features of Go, and remains fully compatible with it. The augmentation is designed to overcome some of the limitations of Go in specific domains:

• Shell scripting, where you want to be able to directly call other executable programs with arguments, without having to navigate all the complexity of the standard os.exec package.

• Numerical / math / data processing, where you want to be able to write simple mathematical expressions operating on vectors, matricies and other more powerful data types, without having to constantly worry about numerical type conversions, and advanced n-dimensional indexing and slicing expressions are critical. Python is the dominant language here precisely because it lets you ignore type information and write such expressions, using operator overloading.

• GPU-based parallel computation, which can greatly speed up some types of parallelizable computations by effectively running many instances of the same code in parallel across a large array of data. The gosl package (automatically run in goal build mode) allows you to run the same Go-based code on a GPU or CPU (using parallel goroutines). See the GPU docs for an overview and comparison to other approaches to GPU computation.

The main goal of Goal is to achieve a "best of both worlds" solution that retains all the type safety and explicitness of Go for all the surrounding control flow and large-scale application logic, while also allowing for a more relaxed syntax in specific, well-defined domains where the Go language has been a barrier. Thus, unlike Python where there are various weak attempts to try to encourage better coding habits, Goal retains in its Go foundation a fundamentally scalable, "industrial strength" language that has already proven its worth in countless real-world applications.

For the shell scripting aspect of Goal, the simple idea is that each line of code is either Go or shell commands, determined in a fairly intuitive way mostly by the content at the start of the line (formal rules below). If a line starts off with something like ls -la... then it is clear that it is not valid Go code, and it is therefore processed as a shell command.

You can intermix Go within a shell line by wrapping an expression with { } braces, and a Go expression can contain shell code by using $. Here's an example:

for i, f := range goalib.SplitLines($ls -la$) {  // ls executes, returns string
    echo {i} {strings.ToLower(f)}              // {} surrounds Go within shell
}

where goalib.SplitLines is a function that runs strings.Split(arg, "\n"), defined in the goalib standard library of such frequently-used helper functions.

For cases where most of the code is standard Go with relatively infrequent use of shell expressions, or in the rare cases where the default interpretation doesn't work, you can explicitly tag a line as shell code using $:

$ chmod +x *.goal

For mathematical expressions, we use # symbols (# = number) to demarcate such expressions. Often you will write entire lines of such expressions:

# x := 1. / (1. + exp(-wts[:, :, :n] * acts[:]))

You can also intermix within Go code:

for _, x := range #[1,2,3]# {
    fmt.Println(#x^2#)
}

Note that you cannot enter math mode directly from shell mode, which is unlikely to be useful anyway (you can wrap in go mode { } if really needed).

In general, the math mode syntax in Goal is designed to be as compatible with Python NumPy / SciPy syntax as possible, while also adding a few Go-specific additions as well: see the Math mode section for details. All elements of a Goal math expression are tensors, which can represent everything from a scalar to an n-dimenstional tensor. These are called "ndarray" in NumPy terms.

The one special form of tensor processing that is available in regular Go code is n dimensional indexing, e.g., tsr[1,2]. This kind of expression with square brackets [ ] and a comma is illegal according to standard Go syntax, so when we detect it, we know that it is being used on a tensor object, and can transpile it into the corresponding tensor.Value or tensor.Set* expression. This is particularly convenient for gosl GPU code that has special support for tensor data. Note that for this GPU use-case, you actually do not want to use math mode, because that engages a different, more complex form of indexing that does not work on the GPU.

The rationale and mnemonics for using $ and # are as follows:

• These are two of the three common ASCII keyboard symbols that are not part of standard Go syntax (@ being the other).

• $ can be thought of as "S" in _S_hell, and is often used for a bash prompt, and many bash examples use it as a prefix. Furthermore, in bash, $( ) is used to wrap shell expressions.

• # is commonly used to refer to numbers. It is also often used as a comment syntax, but on balance the number semantics and uniqueness relative to Go syntax outweigh that issue.

Examples

Here are a few useful examples of Goal code:

You can easily perform handy duration and data size formatting:

22010706 * time.Nanosecond  // 22.010706ms
datasize.Size(44610930)     // 42.5 MB

Shell mode

Environment variables

• set <var> <value> (space delimited as in all shell mode, no equals)

Output redirction

• Standard output redirect: > and >& (and |, |& if needed)

Control flow

• Any error stops the script execution, except for statements wrapped in [ ], indicating an "optional" statement, e.g.:

cd some; [mkdir sub]; cd sub

• & at the end of a statement runs in the background (as in bash) -- otherwise it waits until it completes before it continues.

• jobs, fg, bg, and kill builtin commands function as in usual bash.

Shell functions (aliases)

Use the command keyword to define new functions for Shell mode execution, which can then be used like any other command, for example:

command list {
	ls -la args...
}

cd data
list *.tsv

The command is transpiled into a Go function that takes args ...string. In the command function body, you can use the args... expression to pass all of the args, or args[1] etc to refer to specific positional indexes, as usual.

The command function name is registered so that the standard shell execution code can run the function, passing the args. You can also call it directly from Go code using the standard parentheses expression.

Script Files and Makefile-like functionality

As with most scripting languages, a file of goal code can be made directly executable by appending a "shebang" expression at the start of the file:

#!/usr/bin/env goal

When executed this way, any additional args are available via an args []any variable, which can be passed to a command as follows:

install {args...}

or by referring to specific arg indexes etc.

To make a script behave like a standard Makefile, you can define different commands for each of the make commands, and then add the following at the end of the file to use the args to run commands:

goal.RunCommands(args)

See make for an example, in cmd/goal/testdata/make, which can be run for example using:

./make build

Note that there is nothing special about the name make here, so this can be done with any file.

The make package defines a number of useful utility functions that accomplish the standard dependency and file timestamp checking functionality from the standard make command, as in the magefile system. Note that the goal direct shell command syntax makes the resulting make files much closer to a standard bash-like Makefile, while still having all the benefits of Go control and expressions, compared to magefile.

TODO: implement and document above.

SSH connections to remote hosts

Any number of active SSH connections can be maintained and used dynamically within a script, including simple ways of copying data among the different hosts (including the local host). The Go mode execution is always on the local host in one running process, and only the shell commands are executed remotely, enabling a unique ability to easily coordinate and distribute processing and data across various hosts.

Each host maintains its own working directory and environment variables, which can be configured and re-used by default whenever using a given host.

• gossh hostname.org [name] establishes a connection, using given optional name to refer to this connection. If the name is not provided, a sequential number will be used, starting with 1, with 0 referring always to the local host.

• @name then refers to the given host in all subsequent commands, with @0 referring to the local host where the goal script is running.

• You can use a variable name for the server, like this (the explicit $ $ shell mode is required because a line starting with { is not recognized as shell code):

server := "@myserver"
${server} ls$

Explicit per-command specification of host

@name cd subdir; ls

Default host

@name // or:
gossh @name

uses the given host for all subsequent commands (unless explicitly specified), until the default is changed. Use gossh @0 to return to localhost.

Redirect input / output among hosts

The output of a remote host command can be sent to a file on the local host:

@name cat hostfile.tsv > @0:localfile.tsv

Note the use of the : colon delimiter after the host name here. TODO: You cannot send output to a remote host file (e.g., > @host:remotefile.tsv) -- maybe with sftp?

The output of any command can also be piped to a remote host as its standard input:

ls *.tsv | @host cat > files.txt

scp to copy files easily

The builtin scp function allows easy copying of files across hosts, using the persistent connections established with gossh instead of creating new connections as in the standard scp command.

scp is always run from the local host, with the remote host filename specified as @name:remotefile

scp @name:hostfile.tsv localfile.tsv

Importantly, file wildcard globbing works as expected:

scp @name:*.tsv @0:data/

and entire directories can be copied, as in cp -a or cp -r (this behavior is automatic and does not require a flag).

Close connections

gossh close

Will close all active connections and return the default host to @0. All active connections are also automatically closed when the shell terminates.

Other Utilties

** TODO: need a replacement for findnm -- very powerful but garbage..

Rules for Go vs. Shell determination

These are the rules used to determine whether a line is Go vs. Shell (word = IDENT token):

• $ at the start: Shell.
• Within Shell, {}: Go
• Within Go, $ $: Shell
• Line starts with go keyword: if no ( ) then Shell, else Go
• Line is one word: Shell
• Line starts with path expression (e.g., ./myexec) : Shell
• Line starts with "string": Shell
• Line starts with word word: Shell
• Line starts with word {: Shell
• Otherwise: Go

TODO: update above

Multiple statements per line

• Multiple statements can be combined on one line, separated by ; as in regular Go and shell languages. Critically, the language determination for the first statement determines the language for the remaining statements; you cannot intermix the two on one line, when using ;

Math mode

The math mode in Goal is designed to be generally compatible with Python NumPy / SciPy syntax, so that the widespread experience with that syntax transfers well to Goal. This syntax is also largely compatible with MATLAB and other languages as well. However, we did not fully replicate the NumPy syntax, instead choosing to clean up a few things and generally increase consistency with Go.

In general the Goal global functions are named the same as NumPy, without the np. prefix, which improves readability. It should be very straightforward to write a conversion utility that converts existing NumPy code into Goal code, and that is a better process than trying to make Goal itself perfectly compatible.

All elements of a Goal math expression are tensors (i.e., tensor.Tensor), which can represent everything from a scalar to an n-dimenstional tensor, with different views that support the arbitrary slicing and flexible forms of indexing documented in the table below. These are called an ndarray in NumPy terms. See array vs. tensor NumPy docs for more information. Note that Goal does not have a distinct matrix type; everything is a tensor, and when these are 2D, they function appropriately via the matrix package.

The view versions of Tensor include Sliced, Reshaped, Masked, Indexed, and Rows, each of which wraps around another "source" Tensor, and provides its own way of accessing the underlying data:

• Sliced has an arbitrary set of indexes for each dimension, so access to values along that dimension go through the indexes. Thus, you could reverse the order of the columns (dimension 1), or only operate on a subset of them.

• Masked has a tensor.Bool tensor that filters access to the underlying source tensor through a mask: anywhere the bool value is false, the corresponding source value is not settable, and returns NaN (missing value) when accessed.

• Indexed uses a tensor of indexes where the final, innermost dimension is the same size as the number of dimensions in the wrapped source tensor. The overall shape of this view is that of the remaining outer dimensions of the Indexes tensor, and like other views, assignment and return values are taken from the corresponding indexed value in the wrapped source tensor.

  The current NumPy version of indexed is rather complex and difficult for many people to understand, as articulated in this NEP 21 proposal. The Indexed view at least provides a simpler way of representing the indexes into the source tensor, instead of requiring multiple parallel 1D arrays.

• Rows is an optimized version of Sliced with indexes only for the first, outermost, row dimension.

The following sections provide a full list of equivalents between the tensor Go code, Goal, NumPy, and MATLAB, based on the table in numpy-for-matlab-users.

• The same: in Goal means that the same NumPy syntax works in Goal, minus the np. prefix, and likewise for or: (where Goal also has additional syntax).
• In the tensor.Go code, we sometimes just write a scalar number for simplicity, but these are actually tensor.NewFloat64Scalar etc.
• Goal also has support for string tensors, e.g., for labels, and operators such as addition that make sense for strings are supported. Otherwise, strings are automatically converted to numbers using the tensor.Float interface. If you have any doubt about whether you've got a tensor.Float64 when you expect one, use tensor.AsFloat64Tensor which makes sure.

Tensor shape

Constructing

Ranges and grids

See NumPy docs for details.

Basic indexing

See NumPy basic indexing. Tensor Go uses the Reslice function for all cases (repeated tensor. prefix replaced with t. to take less space). Here you can clearly see the advantage of Goal in allowing significantly more succinct expressions to be written for accomplishing critical tensor functionality.

Boolean tensors and indexing

See NumPy boolean indexing.

Note that Goal only supports boolean logical operators (&& and ||) on boolean tensors, not the single bitwise operators & and |.

Advanced index-based indexing

See NumPy integer indexing. Note that the current NumPy version of indexed is rather complex and difficult for many people to understand, as articulated in this NEP 21 proposal.

TODO: not yet implemented:

Basic math operations (add, multiply, etc)

In Goal and NumPy, the standard +, -, *, / operators perform element-wise operations because those are well-defined for all dimensionalities and are consistent across the different operators, whereas matrix multiplication is specifically used in a 2D linear algebra context, and is not well defined for the other operators.

2D Matrix Linear Algebra

Statistics

FFT and complex numbers

todo: huge amount of work needed to support complex numbers throughout!

tensorfs

The tensorfs data filesystem provides a global filesystem-like workspace for storing tensor data, and Goal has special commands and functions to facilitate interacting with it. In an interactive goal shell, when you do ## to switch into math mode, the prompt changes to show your current directory in the tensorfs, not the regular OS filesystem, and the final prompt character turns into a #.

Use get and set (aliases for tensorfs.Get and tensorfs.Set) to retrieve and store data in the tensorfs:

• x := get("path/to/item") retrieves the tensor data value at given path, which can then be used directly in an expression or saved to a new variable as in this example.

• set("path/to/item", x) saves tensor data to given path, overwriting any existing value for that item if it already exists, and creating a new one if not. x can be any data expression.

You can use the standard shell commands to navigate around the data filesystem:

• cd <dir> to change the current working directory. By default, new variables created in the shell are also recorded into the current working directory for later access.

• ls [-l,r] [dir] list the contents of a directory; without arguments, it shows the current directory. The -l option shows each element on a separate line with its shape. -r does a recursive list through subdirectories.

• mkdir <dir> makes a new subdirectory.

TODO: other commands, etc.