Zakaria/LearnGO
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

whatcanGOwrong

Browse Source
This commit is contained in:
dadgam3er
2024-09-19 21:38:24 -04:00
commit d0ae4d841d
17908 changed files with 4096831 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/.github/workflows/tests.yml
Vendored
+19
View File
@@ -0,0 +1,19 @@
name: Starlark Go Tests
on: [push, pull_request]
jobs:
build:
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
go-version: [1.18.x, 1.19.x, 1.20.x, 1.21.x]
runs-on: ${{ matrix.os }}
steps:
- name: Install Go
uses: actions/setup-go@v4
with:
go-version: ${{ matrix.go-version }}
- name: Checkout code
uses: actions/checkout@v3
- name: Run Tests
shell: bash
run: 'internal/test.sh'
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/CNAME
+1
View File
@@ -0,0 +1 @@
go.starlark.net
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/LICENSE
+29
View File
@@ -0,0 +1,29 @@
Copyright (c) 2017 The Bazel Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the
distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived
from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/README.md
+184
View File
@@ -0,0 +1,184 @@
<!-- This file is the project homepage for go.starlark.net -->
# Starlark in Go
[![Go Tests](https://github.com/google/starlark-go/actions/workflows/tests.yml/badge.svg)](https://github.com/google/starlark-go/actions/workflows/tests.yml)
[![Go Reference](https://pkg.go.dev/badge/go.starlark.net/starlark.svg)](https://pkg.go.dev/go.starlark.net/starlark)
This is the home of the _Starlark in Go_ project.
Starlark in Go is an interpreter for Starlark, implemented in Go.
Starlark was formerly known as Skylark.
The new import path for Go packages is `"go.starlark.net/starlark"`.
Starlark is a dialect of Python intended for use as a configuration language.
Like Python, it is an untyped dynamic language with high-level data
types, first-class functions with lexical scope, and garbage collection.
Unlike CPython, independent Starlark threads execute in parallel, so
Starlark workloads scale well on parallel machines.
Starlark is a small and simple language with a familiar and highly
readable syntax. You can use it as an expressive notation for
structured data, defining functions to eliminate repetition, or you
can use it to add scripting capabilities to an existing application.
A Starlark interpreter is typically embedded within a larger
application, and the application may define additional domain-specific
functions and data types beyond those provided by the core language.
For example, Starlark was originally developed for the
[Bazel build tool](https://bazel.build).
Bazel uses Starlark as the notation both for its BUILD files (like
Makefiles, these declare the executables, libraries, and tests in a
directory) and for [its macro
language](https://docs.bazel.build/versions/master/skylark/language.html),
through which Bazel is extended with custom logic to support new
languages and compilers.
## Documentation
* Language definition: [doc/spec.md](doc/spec.md)
* About the Go implementation: [doc/impl.md](doc/impl.md)
* API documentation: [pkg.go.dev/go.starlark.net/starlark](https://pkg.go.dev/go.starlark.net/starlark)
* Mailing list: [starlark-go](https://groups.google.com/forum/#!forum/starlark-go)
* Issue tracker: [https://github.com/google/starlark-go/issues](https://github.com/google/starlark-go/issues)
### Getting started
Build the code:
```shell
# check out the code and dependencies,
# and install interpreter in $GOPATH/bin
$ go install go.starlark.net/cmd/starlark@latest
```
Run the interpreter:
```console
$ cat coins.star
coins = {
'dime': 10,
'nickel': 5,
'penny': 1,
'quarter': 25,
}
print('By name:\t' + ', '.join(sorted(coins.keys())))
print('By value:\t' + ', '.join(sorted(coins.keys(), key=coins.get)))
$ starlark coins.star
By name: dime, nickel, penny, quarter
By value: penny, nickel, dime, quarter
```
Interact with the read-eval-print loop (REPL):
```pycon
$ starlark
>>> def fibonacci(n):
... res = list(range(n))
... for i in res[2:]:
... res[i] = res[i-2] + res[i-1]
... return res
...
>>> fibonacci(10)
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
>>>
```
When you have finished, type `Ctrl-D` to close the REPL's input stream.
Embed the interpreter in your Go program:
```go
import "go.starlark.net/starlark"
// Execute Starlark program in a file.
thread := &starlark.Thread{Name: "my thread"}
globals, err := starlark.ExecFile(thread, "fibonacci.star", nil, nil)
if err != nil { ... }
// Retrieve a module global.
fibonacci := globals["fibonacci"]
// Call Starlark function from Go.
v, err := starlark.Call(thread, fibonacci, starlark.Tuple{starlark.MakeInt(10)}, nil)
if err != nil { ... }
fmt.Printf("fibonacci(10) = %v\n", v) // fibonacci(10) = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
```
See [starlark/example_test.go](starlark/example_test.go) for more examples.
### Contributing
We welcome submissions but please let us know what you're working on
if you want to change or add to the Starlark repository.
Before undertaking to write something new for the Starlark project,
please file an issue or claim an existing issue.
All significant changes to the language or to the interpreter's Go
API must be discussed before they can be accepted.
This gives all participants a chance to validate the design and to
avoid duplication of effort.
Despite some differences, the Go implementation of Starlark strives to
match the behavior of [the Java implementation](https://github.com/bazelbuild/bazel)
used by Bazel and maintained by the Bazel team.
For that reason, proposals to change the language itself should
generally be directed to [the Starlark site](
https://github.com/bazelbuild/starlark/), not to the maintainers of this
project.
Only once there is consensus that a language change is desirable may
its Go implementation proceed.
We use GitHub pull requests for contributions.
Please complete Google's contributor license agreement (CLA) before
sending your first change to the project. If you are the copyright
holder, you will need to agree to the
[individual contributor license agreement](https://cla.developers.google.com/about/google-individual),
which can be completed online.
If your organization is the copyright holder, the organization will
need to agree to the [corporate contributor license agreement](https://cla.developers.google.com/about/google-corporate).
If the copyright holder for your contribution has already completed
the agreement in connection with another Google open source project,
it does not need to be completed again.
### Stability
We reserve the right to make breaking language and API changes at this
stage in the project, although we will endeavor to keep them to a minimum.
Once the Bazel team has finalized the version 1 language specification,
we will be more rigorous with interface stability.
We aim to support the most recent four (go1.x) releases of the Go
toolchain. For example, if the latest release is go1.20, we support it
along with go1.19, go1.18, and go1.17, but not go1.16.
### Credits
Starlark was designed and implemented in Java by
Jon Brandvein,
Alan Donovan,
Laurent Le Brun,
Dmitry Lomov,
Vladimir Moskva,
François-René Rideau,
Gergely Svigruha, and
Florian Weikert,
standing on the shoulders of the Python community.
The Go implementation was written by Alan Donovan and Jay Conrod;
its scanner was derived from one written by Russ Cox.
### Legal
Starlark in Go is Copyright (c) 2018 The Bazel Authors.
All rights reserved.
It is provided under a 3-clause BSD license:
[LICENSE](https://github.com/google/starlark-go/blob/master/LICENSE).
Starlark in Go is not an official Google product.
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/cmd/starlark/starlark.go
+154
View File
@@ -0,0 +1,154 @@
// Copyright 2017 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// The starlark command interprets a Starlark file.
// With no arguments, it starts a read-eval-print loop (REPL).
package main // import "go.starlark.net/cmd/starlark"
import (
"flag"
"fmt"
"log"
"os"
"runtime"
"runtime/pprof"
"strings"
"go.starlark.net/internal/compile"
"go.starlark.net/lib/json"
"go.starlark.net/lib/math"
"go.starlark.net/lib/time"
"go.starlark.net/repl"
"go.starlark.net/resolve"
"go.starlark.net/starlark"
"golang.org/x/term"
)
// flags
var (
cpuprofile = flag.String("cpuprofile", "", "gather Go CPU profile in this file")
memprofile = flag.String("memprofile", "", "gather Go memory profile in this file")
profile = flag.String("profile", "", "gather Starlark time profile in this file")
showenv = flag.Bool("showenv", false, "on success, print final global environment")
execprog = flag.String("c", "", "execute program `prog`")
)
func init() {
flag.BoolVar(&compile.Disassemble, "disassemble", compile.Disassemble, "show disassembly during compilation of each function")
// non-standard dialect flags
flag.BoolVar(&resolve.AllowSet, "set", resolve.AllowSet, "allow set data type")
flag.BoolVar(&resolve.AllowRecursion, "recursion", resolve.AllowRecursion, "allow while statements and recursive functions")
flag.BoolVar(&resolve.AllowGlobalReassign, "globalreassign", resolve.AllowGlobalReassign, "allow reassignment of globals, and if/for/while statements at top level")
// flags that are now standard
flag.BoolVar(&resolve.AllowFloat, "float", resolve.AllowFloat, "obsolete; no effect")
flag.BoolVar(&resolve.AllowLambda, "lambda", resolve.AllowLambda, "obsolete; no effect")
}
func main() {
os.Exit(doMain())
}
func doMain() int {
log.SetPrefix("starlark: ")
log.SetFlags(0)
flag.Parse()
if *cpuprofile != "" {
f, err := os.Create(*cpuprofile)
check(err)
err = pprof.StartCPUProfile(f)
check(err)
defer func() {
pprof.StopCPUProfile()
err := f.Close()
check(err)
}()
}
if *memprofile != "" {
f, err := os.Create(*memprofile)
check(err)
defer func() {
runtime.GC()
err := pprof.Lookup("heap").WriteTo(f, 0)
check(err)
err = f.Close()
check(err)
}()
}
if *profile != "" {
f, err := os.Create(*profile)
check(err)
err = starlark.StartProfile(f)
check(err)
defer func() {
err := starlark.StopProfile()
check(err)
}()
}
thread := &starlark.Thread{Load: repl.MakeLoad()}
globals := make(starlark.StringDict)
// Ideally this statement would update the predeclared environment.
// TODO(adonovan): plumb predeclared env through to the REPL.
starlark.Universe["json"] = json.Module
starlark.Universe["time"] = time.Module
starlark.Universe["math"] = math.Module
switch {
case flag.NArg() == 1 || *execprog != "":
var (
filename string
src interface{}
err error
)
if *execprog != "" {
// Execute provided program.
filename = "cmdline"
src = *execprog
} else {
// Execute specified file.
filename = flag.Arg(0)
}
thread.Name = "exec " + filename
globals, err = starlark.ExecFile(thread, filename, src, nil)
if err != nil {
repl.PrintError(err)
return 1
}
case flag.NArg() == 0:
stdinIsTerminal := term.IsTerminal(int(os.Stdin.Fd()))
if stdinIsTerminal {
fmt.Println("Welcome to Starlark (go.starlark.net)")
}
thread.Name = "REPL"
repl.REPL(thread, globals)
if stdinIsTerminal {
fmt.Println()
}
default:
log.Print("want at most one Starlark file name")
return 1
}
// Print the global environment.
if *showenv {
for _, name := range globals.Keys() {
if !strings.HasPrefix(name, "_") {
fmt.Fprintf(os.Stderr, "%s = %s\n", name, globals[name])
}
}
}
return 0
}
func check(err error) {
if err != nil {
log.Fatal(err)
}
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/doc/impl.md
+242
View File
@@ -0,0 +1,242 @@
# Starlark in Go: Implementation
This document (a work in progress) describes some of the design
choices of the Go implementation of Starlark.
* [Scanner](#scanner)
* [Parser](#parser)
* [Resolver](#resolver)
* [Evaluator](#evaluator)
* [Data types](#data-types)
* [Freezing](#freezing)
* [Testing](#testing)
## Scanner
The scanner is derived from Russ Cox's
[buildifier](https://github.com/bazelbuild/buildtools/tree/master/buildifier)
tool, which pretty-prints Bazel BUILD files.
Most of the work happens in `(*scanner).nextToken`.
## Parser
The parser is hand-written recursive-descent parser. It uses the
technique of [precedence
climbing](http://www.engr.mun.ca/~theo/Misc/exp_parsing.htm#climbing)
to reduce the number of productions.
In some places the parser accepts a larger set of programs than are
strictly valid, leaving the task of rejecting them to the subsequent
resolver pass. For example, in the function call `f(a, b=c)` the
parser accepts any expression for `a` and `b`, even though `b` may
legally be only an identifier. For the parser to distinguish these
cases would require additional lookahead.
## Resolver
The resolver reports structural errors in the program, such as the use
of `break` and `continue` outside of a loop.
Starlark has stricter syntactic limitations than Python. For example,
it does not permit `for` loops or `if` statements at top level, nor
does it permit global variables to be bound more than once.
These limitations come from the Bazel project's desire to make it easy
to identify the sole statement that defines each global, permitting
accurate cross-reference documentation.
In addition, the resolver validates all variable names, classifying
them as references to universal, global, local, or free variables.
Local and free variables are mapped to a small integer, allowing the
evaluator to use an efficient (flat) representation for the
environment.
Not all features of the Go implementation are "standard" (that is,
supported by Bazel's Java implementation), at least for now, so
non-standard features such as `set`
are flag-controlled. The resolver reports
any uses of dialect features that have not been enabled.
## Evaluator
### Data types
<b>Integers:</b> Integers are representing using `big.Int`, an
arbitrary precision integer. This representation was chosen because,
for many applications, Starlark must be able to handle without loss
protocol buffer values containing signed and unsigned 64-bit integers,
which requires 65 bits of precision.
Small integers (<256) are preallocated, but all other values require
memory allocation. Integer performance is relatively poor, but it
matters little for Bazel-like workloads which depend much
more on lists of strings than on integers. (Recall that a typical loop
over a list in Starlark does not materialize the loop index as an `int`.)
An optimization worth trying would be to represent integers using
either an `int32` or `big.Int`, with the `big.Int` used only when
`int32` does not suffice. Using `int32`, not `int64`, for "small"
numbers would make it easier to detect overflow from operations like
`int32 * int32`, which would trigger the use of `big.Int`.
<b>Floating point</b>:
Floating point numbers are represented using Go's `float64`.
Again, `float` support is required to support protocol buffers. The
existence of floating-point NaN and its infamous comparison behavior
(`NaN != NaN`) had many ramifications for the API, since we cannot
assume the result of an ordered comparison is either less than,
greater than, or equal: it may also fail.
<b>Strings</b>:
TODO: discuss UTF-8 and string.bytes method.
<b>Dictionaries and sets</b>:
Starlark dictionaries have predictable iteration order.
Furthermore, many Starlark values are hashable in Starlark even though
the Go values that represent them are not hashable in Go: big
integers, for example.
Consequently, we cannot use Go maps to implement Starlark's dictionary.
We use a simple hash table whose buckets are linked lists, each
element of which holds up to 8 key/value pairs. In a well-distributed
table the list should rarely exceed length 1. In addition, each
key/value item is part of doubly-linked list that maintains the
insertion order of the elements for iteration.
<b>Struct:</b>
The `starlarkstruct` Go package provides a non-standard Starlark
extension data type, `struct`, that maps field identifiers to
arbitrary values. Fields are accessed using dot notation: `y = s.f`.
This data type is extensively used in Bazel, but its specification is
currently evolving.
Starlark has no `class` mechanism, nor equivalent of Python's
`namedtuple`, though it is likely that future versions will support
some way to define a record data type of several fields, with a
representation more efficient than a hash table.
### Freezing
All mutable values created during module initialization are _frozen_
upon its completion. It is this property that permits a Starlark module
to be referenced by two Starlark threads running concurrently (such as
the initialization threads of two other modules) without the
possibility of a data race.
The Go implementation supports freezing by storing an additional
"frozen" Boolean variable in each mutable object. Once this flag is set,
all subsequent attempts at mutation fail. Every value defines a
Freeze method that sets its own frozen flag if not already set, and
calls Freeze for each value that it contains.
For example, when a list is frozen, it freezes each of its elements;
when a dictionary is frozen, it freezes each of its keys and values;
and when a function value is frozen, it freezes each of the free
variables and parameter default values implicitly referenced by its closure.
Application-defined types must also follow this discipline.
The freeze mechanism in the Go implementation is finer grained than in
the Java implementation: in effect, the latter has one "frozen" flag
per module, and every value holds a reference to the frozen flag of
its module. This makes setting the frozen flag more efficient---a
simple bit flip, no need to traverse the object graph---but coarser
grained. Also, it complicates the API slightly because to construct a
list, say, requires a reference to the frozen flag it should use.
The Go implementation would also permit the freeze operation to be
exposed to the program, for example as a built-in function.
This has proven valuable in writing tests of the freeze mechanism
itself, but is otherwise mostly a curiosity.
### Fail-fast iterators
In some languages (such as Go), a program may mutate a data structure
while iterating over it; for example, a range loop over a map may
delete map elements. In other languages (such as Java), iterators do
extra bookkeeping so that modification of the underlying collection
invalidates the iterator, and the next attempt to use it fails.
This often helps to detect subtle mistakes.
Starlark takes this a step further. Instead of mutation of the
collection invalidating the iterator, the act of iterating makes the
collection temporarily immutable, so that an attempt to, say, delete a
dict element while looping over the dict, will fail. The error is
reported against the delete operation, not the iteration.
This is implemented by having each mutable iterable value record a
counter of active iterators. Starting a loop increments this counter,
and completing a loop decrements it. A collection with a nonzero
counter behaves as if frozen. If the collection is actually frozen,
the counter bookkeeping is unnecessary. (Consequently, iterator
bookkeeping is needed only while objects are still mutable, before
they can have been published to another thread, and thus no
synchronization is necessary.)
A consequence of this design is that in the Go API, it is imperative
to call `Done` on each iterator once it is no longer needed.
```
TODO
starlark.Value interface and subinterfaces
argument passing to builtins: UnpackArgs, UnpackPositionalArgs.
```
<b>Evaluation strategy:</b>
The evaluator uses a simple recursive tree walk, returning a value or
an error for each expression. We have experimented with just-in-time
compilation of syntax trees to bytecode, but two limitations in the
current Go compiler prevent this strategy from outperforming the
tree-walking evaluator.
First, the Go compiler does not generate a "computed goto" for a
switch statement ([Go issue
5496](https://github.com/golang/go/issues/5496)). A bytecode
interpreter's main loop is a for-loop around a switch statement with
dozens or hundreds of cases, and the speed with which each case can be
dispatched strongly affects overall performance.
Currently, a switch statement generates a binary tree of ordered
comparisons, requiring several branches instead of one.
Second, the Go compiler's escape analysis assumes that the underlying
array from a `make([]Value, n)` allocation always escapes
([Go issue 20533](https://github.com/golang/go/issues/20533)).
Because the bytecode interpreter's operand stack has a non-constant
length, it must be allocated with `make`. The resulting allocation
adds to the cost of each Starlark function call; this can be tolerated
by amortizing one very large stack allocation across many calls.
More problematic appears to be the cost of the additional GC write
barriers incurred by every VM operation: every intermediate result is
saved to the VM's operand stack, which is on the heap.
By contrast, intermediate results in the tree-walking evaluator are
never stored to the heap.
```
TODO
frames, backtraces, errors.
threads
Print
Load
```
## Testing
```
TODO
starlarktest package
`assert` module
starlarkstruct
integration with Go testing.T
```
## TODO
```
Discuss practical separation of code and data.
```
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/doc/spec.md
+4483
View File
File diff suppressed because it is too large Load Diff
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/docs/CNAME
+1
View File
@@ -0,0 +1 @@
go.starlark.net
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/docs/cmd/starlark/index.html
+9
View File
@@ -0,0 +1,9 @@
<html>
<head>
<meta name="go-import" content="go.starlark.net git https://github.com/google/starlark-go"></meta>
<meta http-equiv="refresh" content="0;URL='http://godoc.org/go.starlark.net/cmd/starlark'" /></meta>
</head>
<body>
Redirecting to godoc.org page for go.starlark.net/cmd/starlark...
</body>
</html>
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/docs/index.html
+11
View File
@@ -0,0 +1,11 @@
<html>
<!-- This file will be served at go.starlark.net by GitHub pages. -->
<head>
<!-- This tag causes "go get go.starklark.net" to redirect to GitHub. -->
<meta name="go-import" content="go.starlark.net git https://github.com/google/starlark-go"></meta>
<meta http-equiv="refresh" content="0;URL='http://github.com/google/starlark-go'" />
</head>
<body>
Redirecting to GitHub project github.com/google/starlark-go...
</body>
</html>
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/docs/internal/chunkedfile/index.html
+9
View File
@@ -0,0 +1,9 @@
<html>
<head>
<meta name="go-import" content="go.starlark.net git https://github.com/google/starlark-go"></meta>
<meta http-equiv="refresh" content="0;URL='http://godoc.org/go.starlark.net/internal/chunkedfile'" /></meta>
</head>
<body>
Redirecting to godoc.org page for go.starlark.net/internal/chunkedfile...
</body>
</html>
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/docs/internal/compile/index.html
+9
View File
@@ -0,0 +1,9 @@
<html>
<head>
<meta name="go-import" content="go.starlark.net git https://github.com/google/starlark-go"></meta>
<meta http-equiv="refresh" content="0;URL='http://godoc.org/go.starlark.net/internal/compile'" /></meta>
</head>
<body>
Redirecting to godoc.org page for go.starlark.net/internal/compile...
</body>
</html>
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/docs/repl/index.html
+9
View File
@@ -0,0 +1,9 @@
<html>
<head>
<meta name="go-import" content="go.starlark.net git https://github.com/google/starlark-go"></meta>
<meta http-equiv="refresh" content="0;URL='http://godoc.org/go.starlark.net/repl'" /></meta>
</head>
<body>
Redirecting to godoc.org page for go.starlark.net/repl...
</body>
</html>
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/docs/resolve/index.html
+9
View File
@@ -0,0 +1,9 @@
<html>
<head>
<meta name="go-import" content="go.starlark.net git https://github.com/google/starlark-go"></meta>
<meta http-equiv="refresh" content="0;URL='http://godoc.org/go.starlark.net/resolve'" /></meta>
</head>
<body>
Redirecting to godoc.org page for go.starlark.net/resolve...
</body>
</html>
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/docs/starlark/index.html
+9
View File
@@ -0,0 +1,9 @@
<html>
<head>
<meta name="go-import" content="go.starlark.net git https://github.com/google/starlark-go"></meta>
<meta http-equiv="refresh" content="0;URL='http://godoc.org/go.starlark.net/starlark'" /></meta>
</head>
<body>
Redirecting to godoc.org page for go.starlark.net/starlark...
</body>
</html>
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/docs/starlarkstruct/index.html
+9
View File
@@ -0,0 +1,9 @@
<html>
<head>
<meta name="go-import" content="go.starlark.net git https://github.com/google/starlark-go"></meta>
<meta http-equiv="refresh" content="0;URL='http://godoc.org/go.starlark.net/starlarkstruct'" /></meta>
</head>
<body>
Redirecting to godoc.org page for go.starlark.net/starlarkstruct...
</body>
</html>
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/docs/starlarktest/index.html
+9
View File
@@ -0,0 +1,9 @@
<html>
<head>
<meta name="go-import" content="go.starlark.net git https://github.com/google/starlark-go"></meta>
<meta http-equiv="refresh" content="0;URL='http://godoc.org/go.starlark.net/starlarktest'" /></meta>
</head>
<body>
Redirecting to godoc.org page for go.starlark.net/starlarktest...
</body>
</html>
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/docs/syntax/index.html
+9
View File
@@ -0,0 +1,9 @@
<html>
<head>
<meta name="go-import" content="go.starlark.net git https://github.com/google/starlark-go"></meta>
<meta http-equiv="refresh" content="0;URL='http://godoc.org/go.starlark.net/syntax'" /></meta>
</head>
<body>
Redirecting to godoc.org page for go.starlark.net/syntax...
</body>
</html>
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/docs/update.go
+70
View File
@@ -0,0 +1,70 @@
//+build ignore
// The update command creates/updates the <html><head> elements of
// each subpackage beneath docs so that "go get" requests redirect
// to GitHub and other HTTP requests redirect to godoc.corp.
//
// Usage:
//
// $ cd $GOPATH/src/go.starlark.net
// $ go run docs/update.go
//
package main
import (
"bytes"
"fmt"
"log"
"os"
"os/exec"
"path/filepath"
"strings"
)
func main() {
log.SetFlags(0)
log.SetPrefix("update: ")
cwd, err := os.Getwd()
if err != nil {
log.Fatal(err)
}
if filepath.Base(cwd) != "go.starlark.net" {
log.Fatalf("must run from the go.starlark.net directory")
}
cmd := exec.Command("go", "list", "./...")
cmd.Stdout = new(bytes.Buffer)
cmd.Stderr = os.Stderr
if err := cmd.Run(); err != nil {
log.Fatal(err)
}
for _, pkg := range strings.Split(strings.TrimSpace(fmt.Sprint(cmd.Stdout)), "\n") {
rel := strings.TrimPrefix(pkg, "go.starlark.net/") // e.g. "cmd/starlark"
subdir := filepath.Join("docs", rel)
if err := os.MkdirAll(subdir, 0777); err != nil {
log.Fatal(err)
}
// Create missing docs/$rel/index.html files.
html := filepath.Join(subdir, "index.html")
if _, err := os.Stat(html); os.IsNotExist(err) {
data := strings.Replace(defaultHTML, "$PKG", pkg, -1)
if err := os.WriteFile(html, []byte(data), 0666); err != nil {
log.Fatal(err)
}
log.Printf("created %s", html)
}
}
}
const defaultHTML = `<html>
<head>
<meta name="go-import" content="go.starlark.net git https://github.com/google/starlark-go"></meta>
<meta http-equiv="refresh" content="0;URL='http://godoc.org/$PKG'" /></meta>
</head>
<body>
Redirecting to godoc.org page for $PKG...
</body>
</html>
`
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/go.mod
+17
View File
@@ -0,0 +1,17 @@
module go.starlark.net
go 1.18
require (
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e
github.com/google/go-cmp v0.5.1
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8
golang.org/x/term v0.0.0-20220526004731-065cf7ba2467
google.golang.org/protobuf v1.25.0
)
require (
github.com/chzyer/logex v1.1.10 // indirect
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1 // indirect
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
)
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/go.sum
+76
View File
@@ -0,0 +1,76 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/chzyer/logex v1.1.10 h1:Swpa1K6QvQznwJRcfTfQJmTE72DqScAa40E+fbHEXEE=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e h1:fY5BOSpyZCqRo5OhCuC+XN+r/bBCmeuuJtjz+bCNIf8=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1 h1:q763qf9huN11kDQavWsoZXJNW3xEE4JJyHa5Q25/sd8=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.1 h1:JFrFEBb2xKufg6XkJsJr+WbKb4FQlURi5RUcBveYu9k=
github.com/google/go-cmp v0.5.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8 h1:0A+M6Uqn+Eje4kHMK80dtF3JCXC4ykBgQG4Fe06QRhQ=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20220526004731-065cf7ba2467 h1:CBpWXWQpIRjzmkkA+M7q9Fqnwd2mZr3AFqexg8YTfoM=
golang.org/x/term v0.0.0-20220526004731-065cf7ba2467/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.25.0 h1:Ejskq+SyPohKW+1uil0JJMtmHCgJPJ/qWTxr8qp+R4c=
google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/internal/chunkedfile/chunkedfile.go
+131
View File
@@ -0,0 +1,131 @@
// Copyright 2017 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package chunkedfile provides utilities for testing that source code
// errors are reported in the appropriate places.
//
// A chunked file consists of several chunks of input text separated by
// "---" lines. Each chunk is an input to the program under test, such
// as an evaluator. Lines containing "###" are interpreted as
// expectations of failure: the following text is a Go string literal
// denoting a regular expression that should match the failure message.
//
// Example:
//
// x = 1 / 0 ### "division by zero"
// ---
// x = 1
// print(x + "") ### "int + string not supported"
//
// A client test feeds each chunk of text into the program under test,
// then calls chunk.GotError for each error that actually occurred. Any
// discrepancy between the actual and expected errors is reported using
// the client's reporter, which is typically a testing.T.
package chunkedfile // import "go.starlark.net/internal/chunkedfile"
import (
"fmt"
"os"
"regexp"
"runtime"
"strconv"
"strings"
)
const debug = false
// A Chunk is a portion of a source file.
// It contains a set of expected errors.
type Chunk struct {
Source string
filename string
report Reporter
wantErrs map[int]*regexp.Regexp
}
// Reporter is implemented by *testing.T.
type Reporter interface {
Errorf(format string, args ...interface{})
}
// Read parses a chunked file and returns its chunks.
// It reports failures using the reporter.
//
// Error messages of the form "file.star:line:col: ..." are prefixed
// by a newline so that the Go source position added by (*testing.T).Errorf
// appears on a separate line so as not to confused editors.
func Read(filename string, report Reporter) (chunks []Chunk) {
data, err := os.ReadFile(filename)
if err != nil {
report.Errorf("%s", err)
return
}
linenum := 1
eol := "\n"
if runtime.GOOS == "windows" {
eol = "\r\n"
}
for i, chunk := range strings.Split(string(data), eol+"---"+eol) {
if debug {
fmt.Printf("chunk %d at line %d: %s\n", i, linenum, chunk)
}
// Pad with newlines so the line numbers match the original file.
src := strings.Repeat("\n", linenum-1) + chunk
wantErrs := make(map[int]*regexp.Regexp)
// Parse comments of the form:
// ### "expected error".
lines := strings.Split(chunk, "\n")
for j := 0; j < len(lines); j, linenum = j+1, linenum+1 {
line := lines[j]
hashes := strings.Index(line, "###")
if hashes < 0 {
continue
}
rest := strings.TrimSpace(line[hashes+len("###"):])
pattern, err := strconv.Unquote(rest)
if err != nil {
report.Errorf("\n%s:%d: not a quoted regexp: %s", filename, linenum, rest)
continue
}
rx, err := regexp.Compile(pattern)
if err != nil {
report.Errorf("\n%s:%d: %v", filename, linenum, err)
continue
}
wantErrs[linenum] = rx
if debug {
fmt.Printf("\t%d\t%s\n", linenum, rx)
}
}
linenum++
chunks = append(chunks, Chunk{src, filename, report, wantErrs})
}
return chunks
}
// GotError should be called by the client to report an error at a particular line.
// GotError reports unexpected errors to the chunk's reporter.
func (chunk *Chunk) GotError(linenum int, msg string) {
if rx, ok := chunk.wantErrs[linenum]; ok {
delete(chunk.wantErrs, linenum)
if !rx.MatchString(msg) {
chunk.report.Errorf("\n%s:%d: error %q does not match pattern %q", chunk.filename, linenum, msg, rx)
}
} else {
chunk.report.Errorf("\n%s:%d: unexpected error: %v", chunk.filename, linenum, msg)
}
}
// Done should be called by the client to indicate that the chunk has no more errors.
// Done reports expected errors that did not occur to the chunk's reporter.
func (chunk *Chunk) Done() {
for linenum, rx := range chunk.wantErrs {
chunk.report.Errorf("\n%s:%d: expected error matching %q", chunk.filename, linenum, rx)
}
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/internal/compile/codegen_test.go
+118
View File
@@ -0,0 +1,118 @@
package compile
import (
"bytes"
"fmt"
"testing"
"go.starlark.net/resolve"
"go.starlark.net/syntax"
)
// TestPlusFolding ensures that the compiler generates optimized code for
// n-ary addition of strings, lists, and tuples.
func TestPlusFolding(t *testing.T) {
isPredeclared := func(name string) bool { return name == "x" }
isUniversal := func(name string) bool { return false }
for i, test := range []struct {
src string // source expression
want string // disassembled code
}{
{
// string folding
`"a" + "b" + "c" + "d"`,
`constant "abcd"; return`,
},
{
// string folding with variable:
`"a" + "b" + x + "c" + "d"`,
`constant "ab"; predeclared x; plus; constant "cd"; plus; return`,
},
{
// list folding
`[1] + [2] + [3]`,
`constant 1; constant 2; constant 3; makelist<3>; return`,
},
{
// list folding with variable
`[1] + [2] + x + [3]`,
`constant 1; constant 2; makelist<2>; ` +
`predeclared x; plus; ` +
`constant 3; makelist<1>; plus; ` +
`return`,
},
{
// tuple folding
`() + (1,) + (2, 3)`,
`constant 1; constant 2; constant 3; maketuple<3>; return`,
},
{
// tuple folding with variable
`() + (1,) + x + (2, 3)`,
`constant 1; maketuple<1>; predeclared x; plus; ` +
`constant 2; constant 3; maketuple<2>; plus; ` +
`return`,
},
} {
expr, err := syntax.ParseExpr("in.star", test.src, 0)
if err != nil {
t.Errorf("#%d: %v", i, err)
continue
}
locals, err := resolve.Expr(expr, isPredeclared, isUniversal)
if err != nil {
t.Errorf("#%d: %v", i, err)
continue
}
got := disassemble(Expr(syntax.LegacyFileOptions(), expr, "<expr>", locals).Toplevel)
if test.want != got {
t.Errorf("expression <<%s>> generated <<%s>>, want <<%s>>",
test.src, got, test.want)
}
}
}
// disassemble is a trivial disassembler tailored to the accumulator test.
func disassemble(f *Funcode) string {
out := new(bytes.Buffer)
code := f.Code
for pc := 0; pc < len(code); {
op := Opcode(code[pc])
pc++
// TODO(adonovan): factor in common with interpreter.
var arg uint32
if op >= OpcodeArgMin {
for s := uint(0); ; s += 7 {
b := code[pc]
pc++
arg |= uint32(b&0x7f) << s
if b < 0x80 {
break
}
}
}
if out.Len() > 0 {
out.WriteString("; ")
}
fmt.Fprintf(out, "%s", op)
if op >= OpcodeArgMin {
switch op {
case CONSTANT:
switch x := f.Prog.Constants[arg].(type) {
case string:
fmt.Fprintf(out, " %q", x)
default:
fmt.Fprintf(out, " %v", x)
}
case LOCAL:
fmt.Fprintf(out, " %s", f.Locals[arg].Name)
case PREDECLARED:
fmt.Fprintf(out, " %s", f.Prog.Names[arg])
default:
fmt.Fprintf(out, "<%d>", arg)
}
}
}
return out.String()
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/internal/compile/compile.go
+1927
View File
File diff suppressed because it is too large Load Diff
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/internal/compile/compile_test.go
+74
View File
@@ -0,0 +1,74 @@
package compile_test
import (
"bytes"
"strings"
"testing"
"go.starlark.net/starlark"
)
// TestSerialization verifies that a serialized program can be loaded,
// deserialized, and executed.
func TestSerialization(t *testing.T) {
predeclared := starlark.StringDict{
"x": starlark.String("mur"),
"n": starlark.MakeInt(2),
}
const src = `
def mul(a, b):
return a * b
y = mul(x, n)
`
_, oldProg, err := starlark.SourceProgram("mul.star", src, predeclared.Has)
if err != nil {
t.Fatal(err)
}
buf := new(bytes.Buffer)
if err := oldProg.Write(buf); err != nil {
t.Fatalf("oldProg.WriteTo: %v", err)
}
newProg, err := starlark.CompiledProgram(buf)
if err != nil {
t.Fatalf("CompiledProgram: %v", err)
}
thread := new(starlark.Thread)
globals, err := newProg.Init(thread, predeclared)
if err != nil {
t.Fatalf("newProg.Init: %v", err)
}
if got, want := globals["y"], starlark.String("murmur"); got != want {
t.Errorf("Value of global was %s, want %s", got, want)
t.Logf("globals: %v", globals)
}
// Verify stack frame.
predeclared["n"] = starlark.None
_, err = newProg.Init(thread, predeclared)
evalErr, ok := err.(*starlark.EvalError)
if !ok {
t.Fatalf("newProg.Init call returned err %v, want *EvalError", err)
}
const want = `Traceback (most recent call last):
mul.star:5:8: in <toplevel>
mul.star:3:14: in mul
Error: unknown binary op: string * NoneType`
if got := evalErr.Backtrace(); got != want {
t.Fatalf("got <<%s>>, want <<%s>>", got, want)
}
}
func TestGarbage(t *testing.T) {
const garbage = "This is not a compiled Starlark program."
_, err := starlark.CompiledProgram(strings.NewReader(garbage))
if err == nil {
t.Fatalf("CompiledProgram did not report an error when decoding garbage")
}
if !strings.Contains(err.Error(), "not a compiled module") {
t.Fatalf("CompiledProgram reported the wrong error when decoding garbage: %v", err)
}
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/internal/compile/serial.go
+399
View File
@@ -0,0 +1,399 @@
package compile
// This file defines functions to read and write a compile.Program to a file.
//
// It is the client's responsibility to avoid version skew between the
// compiler used to produce a file and the interpreter that consumes it.
// The version number is provided as a constant.
// Incompatible protocol changes should also increment the version number.
//
// Encoding
//
// Program:
// "sky!" [4]byte # magic number
// str uint32le # offset of <strings> section
// version varint # must match Version
// filename string
// numloads varint
// loads []Ident
// numnames varint
// names []string
// numconsts varint
// consts []Constant
// numglobals varint
// globals []Ident
// toplevel Funcode
// numfuncs varint
// funcs []Funcode
// recursion varint (0 or 1)
// <strings> []byte # concatenation of all referenced strings
// EOF
//
// Funcode:
// id Ident
// code []byte
// pclinetablen varint
// pclinetab []varint
// numlocals varint
// locals []Ident
// numcells varint
// cells []int
// numfreevars varint
// freevar []Ident
// maxstack varint
// numparams varint
// numkwonlyparams varint
// hasvarargs varint (0 or 1)
// haskwargs varint (0 or 1)
//
// Ident:
// filename string
// line, col varint
//
// Constant: # type data
// type varint # 0=string string
// data ... # 1=bytes string
// # 2=int varint
// # 3=float varint (bits as uint64)
// # 4=bigint string (decimal ASCII text)
//
// The encoding starts with a four-byte magic number.
// The next four bytes are a little-endian uint32
// that provides the offset of the string section
// at the end of the file, which contains the ordered
// concatenation of all strings referenced by the
// program. This design permits the decoder to read
// the first and second parts of the file into different
// memory allocations: the first (the encoded program)
// is transient, but the second (the strings) persists
// for the life of the Program.
//
// Within the encoded program, all strings are referred
// to by their length. As the encoder and decoder process
// the entire file sequentially, they are in lock step,
// so the start offset of each string is implicit.
//
// Program.Code is represented as a []byte slice to permit
// modification when breakpoints are set. All other strings
// are represented as strings. They all (unsafely) share the
// same backing byte slice.
//
// Aside from the str field, all integers are encoded as varints.
import (
"encoding/binary"
"fmt"
"math"
"math/big"
debugpkg "runtime/debug"
"unsafe"
"go.starlark.net/syntax"
)
const magic = "!sky"
// Encode encodes a compiled Starlark program.
func (prog *Program) Encode() []byte {
var e encoder
e.p = append(e.p, magic...)
e.p = append(e.p, "????"...) // string data offset; filled in later
e.int(Version)
e.string(prog.Toplevel.Pos.Filename())
e.bindings(prog.Loads)
e.int(len(prog.Names))
for _, name := range prog.Names {
e.string(name)
}
e.int(len(prog.Constants))
for _, c := range prog.Constants {
switch c := c.(type) {
case string:
e.int(0)
e.string(c)
case Bytes:
e.int(1)
e.string(string(c))
case int64:
e.int(2)
e.int64(c)
case float64:
e.int(3)
e.uint64(math.Float64bits(c))
case *big.Int:
e.int(4)
e.string(c.Text(10))
}
}
e.bindings(prog.Globals)
e.function(prog.Toplevel)
e.int(len(prog.Functions))
for _, fn := range prog.Functions {
e.function(fn)
}
e.int(b2i(prog.Recursion))
// Patch in the offset of the string data section.
binary.LittleEndian.PutUint32(e.p[4:8], uint32(len(e.p)))
return append(e.p, e.s...)
}
type encoder struct {
p []byte // encoded program
s []byte // strings
tmp [binary.MaxVarintLen64]byte
}
func (e *encoder) int(x int) {
e.int64(int64(x))
}
func (e *encoder) int64(x int64) {
n := binary.PutVarint(e.tmp[:], x)
e.p = append(e.p, e.tmp[:n]...)
}
func (e *encoder) uint64(x uint64) {
n := binary.PutUvarint(e.tmp[:], x)
e.p = append(e.p, e.tmp[:n]...)
}
func (e *encoder) string(s string) {
e.int(len(s))
e.s = append(e.s, s...)
}
func (e *encoder) bytes(b []byte) {
e.int(len(b))
e.s = append(e.s, b...)
}
func (e *encoder) binding(bind Binding) {
e.string(bind.Name)
e.int(int(bind.Pos.Line))
e.int(int(bind.Pos.Col))
}
func (e *encoder) bindings(binds []Binding) {
e.int(len(binds))
for _, bind := range binds {
e.binding(bind)
}
}
func (e *encoder) function(fn *Funcode) {
e.binding(Binding{fn.Name, fn.Pos})
e.string(fn.Doc)
e.bytes(fn.Code)
e.int(len(fn.pclinetab))
for _, x := range fn.pclinetab {
e.int64(int64(x))
}
e.bindings(fn.Locals)
e.int(len(fn.Cells))
for _, index := range fn.Cells {
e.int(index)
}
e.bindings(fn.Freevars)
e.int(fn.MaxStack)
e.int(fn.NumParams)
e.int(fn.NumKwonlyParams)
e.int(b2i(fn.HasVarargs))
e.int(b2i(fn.HasKwargs))
}
func b2i(b bool) int {
if b {
return 1
} else {
return 0
}
}
// DecodeProgram decodes a compiled Starlark program from data.
func DecodeProgram(data []byte) (_ *Program, err error) {
if len(data) < len(magic) {
return nil, fmt.Errorf("not a compiled module: no magic number")
}
if got := string(data[:4]); got != magic {
return nil, fmt.Errorf("not a compiled module: got magic number %q, want %q",
got, magic)
}
defer func() {
if x := recover(); x != nil {
debugpkg.PrintStack()
err = fmt.Errorf("internal error while decoding program: %v", x)
}
}()
offset := binary.LittleEndian.Uint32(data[4:8])
d := decoder{
p: data[8:offset],
s: append([]byte(nil), data[offset:]...), // allocate a copy, which will persist
}
if v := d.int(); v != Version {
return nil, fmt.Errorf("version mismatch: read %d, want %d", v, Version)
}
filename := d.string()
d.filename = &filename
loads := d.bindings()
names := make([]string, d.int())
for i := range names {
names[i] = d.string()
}
// constants
constants := make([]interface{}, d.int())
for i := range constants {
var c interface{}
switch d.int() {
case 0:
c = d.string()
case 1:
c = Bytes(d.string())
case 2:
c = d.int64()
case 3:
c = math.Float64frombits(d.uint64())
case 4:
c, _ = new(big.Int).SetString(d.string(), 10)
}
constants[i] = c
}
globals := d.bindings()
toplevel := d.function()
funcs := make([]*Funcode, d.int())
for i := range funcs {
funcs[i] = d.function()
}
recursion := d.int() != 0
prog := &Program{
Loads: loads,
Names: names,
Constants: constants,
Globals: globals,
Functions: funcs,
Toplevel: toplevel,
Recursion: recursion,
}
toplevel.Prog = prog
for _, f := range funcs {
f.Prog = prog
}
if len(d.p)+len(d.s) > 0 {
return nil, fmt.Errorf("internal error: unconsumed data during decoding")
}
return prog, nil
}
type decoder struct {
p []byte // encoded program
s []byte // strings
filename *string // (indirect to avoid keeping decoder live)
}
func (d *decoder) int() int {
return int(d.int64())
}
func (d *decoder) int64() int64 {
x, len := binary.Varint(d.p[:])
d.p = d.p[len:]
return x
}
func (d *decoder) uint64() uint64 {
x, len := binary.Uvarint(d.p[:])
d.p = d.p[len:]
return x
}
func (d *decoder) string() (s string) {
if slice := d.bytes(); len(slice) > 0 {
// Avoid a memory allocation for each string
// by unsafely aliasing slice.
type string struct {
data *byte
len int
}
ptr := (*string)(unsafe.Pointer(&s))
ptr.data = &slice[0]
ptr.len = len(slice)
}
return s
}
func (d *decoder) bytes() []byte {
len := d.int()
r := d.s[:len:len]
d.s = d.s[len:]
return r
}
func (d *decoder) binding() Binding {
name := d.string()
line := int32(d.int())
col := int32(d.int())
return Binding{Name: name, Pos: syntax.MakePosition(d.filename, line, col)}
}
func (d *decoder) bindings() []Binding {
bindings := make([]Binding, d.int())
for i := range bindings {
bindings[i] = d.binding()
}
return bindings
}
func (d *decoder) ints() []int {
ints := make([]int, d.int())
for i := range ints {
ints[i] = d.int()
}
return ints
}
func (d *decoder) bool() bool { return d.int() != 0 }
func (d *decoder) function() *Funcode {
id := d.binding()
doc := d.string()
code := d.bytes()
pclinetab := make([]uint16, d.int())
for i := range pclinetab {
pclinetab[i] = uint16(d.int())
}
locals := d.bindings()
cells := d.ints()
freevars := d.bindings()
maxStack := d.int()
numParams := d.int()
numKwonlyParams := d.int()
hasVarargs := d.int() != 0
hasKwargs := d.int() != 0
return &Funcode{
// Prog is filled in later.
Pos: id.Pos,
Name: id.Name,
Doc: doc,
Code: code,
pclinetab: pclinetab,
Locals: locals,
Cells: cells,
Freevars: freevars,
MaxStack: maxStack,
NumParams: numParams,
NumKwonlyParams: numKwonlyParams,
HasVarargs: hasVarargs,
HasKwargs: hasKwargs,
}
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/internal/spell/spell.go
+115
View File
@@ -0,0 +1,115 @@
// Package spell file defines a simple spelling checker for use in attribute errors
// such as "no such field .foo; did you mean .food?".
package spell
import (
"strings"
"unicode"
)
// Nearest returns the element of candidates
// nearest to x using the Levenshtein metric,
// or "" if none were promising.
func Nearest(x string, candidates []string) string {
// Ignore underscores and case when matching.
fold := func(s string) string {
return strings.Map(func(r rune) rune {
if r == '_' {
return -1
}
return unicode.ToLower(r)
}, s)
}
x = fold(x)
var best string
bestD := (len(x) + 1) / 2 // allow up to 50% typos
for _, c := range candidates {
d := levenshtein(x, fold(c), bestD)
if d < bestD {
bestD = d
best = c
}
}
return best
}
// levenshtein returns the non-negative Levenshtein edit distance
// between the byte strings x and y.
//
// If the computed distance exceeds max,
// the function may return early with an approximate value > max.
func levenshtein(x, y string, max int) int {
// This implementation is derived from one by Laurent Le Brun in
// Bazel that uses the single-row space efficiency trick
// described at bitbucket.org/clearer/iosifovich.
// Let x be the shorter string.
if len(x) > len(y) {
x, y = y, x
}
// Remove common prefix.
for i := 0; i < len(x); i++ {
if x[i] != y[i] {
x = x[i:]
y = y[i:]
break
}
}
if x == "" {
return len(y)
}
if d := abs(len(x) - len(y)); d > max {
return d // excessive length divergence
}
row := make([]int, len(y)+1)
for i := range row {
row[i] = i
}
for i := 1; i <= len(x); i++ {
row[0] = i
best := i
prev := i - 1
for j := 1; j <= len(y); j++ {
a := prev + b2i(x[i-1] != y[j-1]) // substitution
b := 1 + row[j-1] // deletion
c := 1 + row[j] // insertion
k := min(a, min(b, c))
prev, row[j] = row[j], k
best = min(best, k)
}
if best > max {
return best
}
}
return row[len(y)]
}
func b2i(b bool) int {
if b {
return 1
} else {
return 0
}
}
func min(x, y int) int {
if x < y {
return x
} else {
return y
}
}
func abs(x int) int {
if x >= 0 {
return x
} else {
return -x
}
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/internal/test.sh
+19
View File
@@ -0,0 +1,19 @@
#!/bin/sh
# Copyright 2021 The Bazel Authors. All rights reserved.
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file.
set -eu
# Confirm that go.mod and go.sum are tidy.
cp go.mod go.mod.orig
cp go.sum go.sum.orig
go mod tidy
# Use -w to ignore differences in OS newlines.
diff -w go.mod.orig go.mod || { echo "go.mod is not tidy"; exit 1; }
diff -w go.sum.orig go.sum || { echo "go.sum is not tidy"; exit 1; }
rm go.mod.orig go.sum.orig
# Run tests
go test ./...
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/lib/json/json.go
+537
View File
@@ -0,0 +1,537 @@
// Copyright 2020 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package json defines utilities for converting Starlark values
// to/from JSON strings. The most recent IETF standard for JSON is
// https://www.ietf.org/rfc/rfc7159.txt.
package json // import "go.starlark.net/lib/json"
import (
"bytes"
"encoding/json"
"fmt"
"math"
"math/big"
"reflect"
"sort"
"strconv"
"strings"
"unicode/utf8"
"unsafe"
"go.starlark.net/starlark"
"go.starlark.net/starlarkstruct"
)
// Module json is a Starlark module of JSON-related functions.
//
// json = module(
// encode,
// decode,
// indent,
// )
//
// def encode(x):
//
// The encode function accepts one required positional argument,
// which it converts to JSON by cases:
// - A Starlark value that implements Go's standard json.Marshal
// interface defines its own JSON encoding.
// - None, True, and False are converted to null, true, and false, respectively.
// - Starlark int values, no matter how large, are encoded as decimal integers.
// Some decoders may not be able to decode very large integers.
// - Starlark float values are encoded using decimal point notation,
// even if the value is an integer.
// It is an error to encode a non-finite floating-point value.
// - Starlark strings are encoded as JSON strings, using UTF-16 escapes.
// - a Starlark IterableMapping (e.g. dict) is encoded as a JSON object.
// It is an error if any key is not a string.
// - any other Starlark Iterable (e.g. list, tuple) is encoded as a JSON array.
// - a Starlark HasAttrs (e.g. struct) is encoded as a JSON object.
//
// It an application-defined type matches more than one the cases describe above,
// (e.g. it implements both Iterable and HasFields), the first case takes precedence.
// Encoding any other value yields an error.
//
// def decode(x[, default]):
//
// The decode function has one required positional parameter, a JSON string.
// It returns the Starlark value that the string denotes.
// - Numbers are parsed as int or float, depending on whether they
// contain a decimal point.
// - JSON objects are parsed as new unfrozen Starlark dicts.
// - JSON arrays are parsed as new unfrozen Starlark lists.
//
// If x is not a valid JSON string, the behavior depends on the "default"
// parameter: if present, Decode returns its value; otherwise, Decode fails.
//
// def indent(str, *, prefix="", indent="\t"):
//
// The indent function pretty-prints a valid JSON encoding,
// and returns a string containing the indented form.
// It accepts one required positional parameter, the JSON string,
// and two optional keyword-only string parameters, prefix and indent,
// that specify a prefix of each new line, and the unit of indentation.
var Module = &starlarkstruct.Module{
Name: "json",
Members: starlark.StringDict{
"encode": starlark.NewBuiltin("json.encode", encode),
"decode": starlark.NewBuiltin("json.decode", decode),
"indent": starlark.NewBuiltin("json.indent", indent),
},
}
func encode(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var x starlark.Value
if err := starlark.UnpackPositionalArgs(b.Name(), args, kwargs, 1, &x); err != nil {
return nil, err
}
buf := new(bytes.Buffer)
var quoteSpace [128]byte
quote := func(s string) {
// Non-trivial escaping is handled by Go's encoding/json.
if isPrintableASCII(s) {
buf.Write(strconv.AppendQuote(quoteSpace[:0], s))
} else {
// TODO(adonovan): opt: RFC 8259 mandates UTF-8 for JSON.
// Can we avoid this call?
data, _ := json.Marshal(s)
buf.Write(data)
}
}
path := make([]unsafe.Pointer, 0, 8)
var emit func(x starlark.Value) error
emit = func(x starlark.Value) error {
// It is only necessary to push/pop the item when it might contain
// itself (i.e. the last three switch cases), but omitting it in the other
// cases did not show significant improvement on the benchmarks.
if ptr := pointer(x); ptr != nil {
if pathContains(path, ptr) {
return fmt.Errorf("cycle in JSON structure")
}
path = append(path, ptr)
defer func() { path = path[0 : len(path)-1] }()
}
switch x := x.(type) {
case json.Marshaler:
// Application-defined starlark.Value types
// may define their own JSON encoding.
data, err := x.MarshalJSON()
if err != nil {
return err
}
buf.Write(data)
case starlark.NoneType:
buf.WriteString("null")
case starlark.Bool:
if x {
buf.WriteString("true")
} else {
buf.WriteString("false")
}
case starlark.Int:
fmt.Fprint(buf, x)
case starlark.Float:
if !isFinite(float64(x)) {
return fmt.Errorf("cannot encode non-finite float %v", x)
}
fmt.Fprintf(buf, "%g", x) // always contains a decimal point
case starlark.String:
quote(string(x))
case starlark.IterableMapping:
// e.g. dict (must have string keys)
buf.WriteByte('{')
items := x.Items()
for _, item := range items {
if _, ok := item[0].(starlark.String); !ok {
return fmt.Errorf("%s has %s key, want string", x.Type(), item[0].Type())
}
}
sort.Slice(items, func(i, j int) bool {
return items[i][0].(starlark.String) < items[j][0].(starlark.String)
})
for i, item := range items {
if i > 0 {
buf.WriteByte(',')
}
k, _ := starlark.AsString(item[0])
quote(k)
buf.WriteByte(':')
if err := emit(item[1]); err != nil {
return fmt.Errorf("in %s key %s: %v", x.Type(), item[0], err)
}
}
buf.WriteByte('}')
case starlark.Iterable:
// e.g. tuple, list
buf.WriteByte('[')
iter := x.Iterate()
defer iter.Done()
var elem starlark.Value
for i := 0; iter.Next(&elem); i++ {
if i > 0 {
buf.WriteByte(',')
}
if err := emit(elem); err != nil {
return fmt.Errorf("at %s index %d: %v", x.Type(), i, err)
}
}
buf.WriteByte(']')
case starlark.HasAttrs:
// e.g. struct
buf.WriteByte('{')
var names []string
names = append(names, x.AttrNames()...)
sort.Strings(names)
for i, name := range names {
v, err := x.Attr(name)
if err != nil {
return fmt.Errorf("cannot access attribute %s.%s: %w", x.Type(), name, err)
}
if v == nil {
// x.AttrNames() returned name, but x.Attr(name) returned nil, stating
// that the field doesn't exist.
return fmt.Errorf("missing attribute %s.%s (despite %q appearing in dir()", x.Type(), name, name)
}
if i > 0 {
buf.WriteByte(',')
}
quote(name)
buf.WriteByte(':')
if err := emit(v); err != nil {
return fmt.Errorf("in field .%s: %v", name, err)
}
}
buf.WriteByte('}')
default:
return fmt.Errorf("cannot encode %s as JSON", x.Type())
}
return nil
}
if err := emit(x); err != nil {
return nil, fmt.Errorf("%s: %v", b.Name(), err)
}
return starlark.String(buf.String()), nil
}
func pointer(i interface{}) unsafe.Pointer {
v := reflect.ValueOf(i)
switch v.Kind() {
case reflect.Ptr, reflect.Chan, reflect.Map, reflect.UnsafePointer, reflect.Slice:
// TODO(adonovan): use v.Pointer() when we drop go1.17.
return unsafe.Pointer(v.Pointer())
default:
return nil
}
}
func pathContains(path []unsafe.Pointer, item unsafe.Pointer) bool {
for _, p := range path {
if p == item {
return true
}
}
return false
}
// isPrintableASCII reports whether s contains only printable ASCII.
func isPrintableASCII(s string) bool {
for i := 0; i < len(s); i++ {
b := s[i]
if b < 0x20 || b >= 0x80 {
return false
}
}
return true
}
// isFinite reports whether f represents a finite rational value.
// It is equivalent to !math.IsNan(f) && !math.IsInf(f, 0).
func isFinite(f float64) bool {
return math.Abs(f) <= math.MaxFloat64
}
func indent(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
prefix, indent := "", "\t" // keyword-only
if err := starlark.UnpackArgs(b.Name(), nil, kwargs,
"prefix?", &prefix,
"indent?", &indent,
); err != nil {
return nil, err
}
var str string // positional-only
if err := starlark.UnpackPositionalArgs(b.Name(), args, nil, 1, &str); err != nil {
return nil, err
}
buf := new(bytes.Buffer)
if err := json.Indent(buf, []byte(str), prefix, indent); err != nil {
return nil, fmt.Errorf("%s: %v", b.Name(), err)
}
return starlark.String(buf.String()), nil
}
func decode(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (v starlark.Value, err error) {
var s string
var d starlark.Value
if err := starlark.UnpackArgs(b.Name(), args, kwargs, "x", &s, "default?", &d); err != nil {
return nil, err
}
if len(args) < 1 {
// "x" parameter is positional only; UnpackArgs does not allow us to
// directly express "def decode(x, *, default)"
return nil, fmt.Errorf("%s: unexpected keyword argument x", b.Name())
}
// The decoder necessarily makes certain representation choices
// such as list vs tuple, struct vs dict, int vs float.
// In principle, we could parameterize it to allow the caller to
// control the returned types, but there's no compelling need yet.
// Use panic/recover with a distinguished type (failure) for error handling.
// If "default" is set, we only want to return it when encountering invalid
// json - not for any other possible causes of panic.
// In particular, if we ever extend the json.decode API to take a callback,
// a distinguished, private failure type prevents the possibility of
// json.decode with "default" becoming abused as a try-catch mechanism.
type failure string
fail := func(format string, args ...interface{}) {
panic(failure(fmt.Sprintf(format, args...)))
}
i := 0
// skipSpace consumes leading spaces, and reports whether there is more input.
skipSpace := func() bool {
for ; i < len(s); i++ {
b := s[i]
if b != ' ' && b != '\t' && b != '\n' && b != '\r' {
return true
}
}
return false
}
// next consumes leading spaces and returns the first non-space.
// It panics if at EOF.
next := func() byte {
if skipSpace() {
return s[i]
}
fail("unexpected end of file")
panic("unreachable")
}
// parse returns the next JSON value from the input.
// It consumes leading but not trailing whitespace.
// It panics on error.
var parse func() starlark.Value
parse = func() starlark.Value {
b := next()
switch b {
case '"':
// string
// Find end of quotation.
// Also, record whether trivial unquoting is safe.
// Non-trivial unquoting is handled by Go's encoding/json.
safe := true
closed := false
j := i + 1
for ; j < len(s); j++ {
b := s[j]
if b == '\\' {
safe = false
j++ // skip x in \x
} else if b == '"' {
closed = true
j++ // skip '"'
break
} else if b >= utf8.RuneSelf {
safe = false
}
}
if !closed {
fail("unclosed string literal")
}
r := s[i:j]
i = j
// unquote
if safe {
r = r[1 : len(r)-1]
} else if err := json.Unmarshal([]byte(r), &r); err != nil {
fail("%s", err)
}
return starlark.String(r)
case 'n':
if strings.HasPrefix(s[i:], "null") {
i += len("null")
return starlark.None
}
case 't':
if strings.HasPrefix(s[i:], "true") {
i += len("true")
return starlark.True
}
case 'f':
if strings.HasPrefix(s[i:], "false") {
i += len("false")
return starlark.False
}
case '[':
// array
var elems []starlark.Value
i++ // '['
b = next()
if b != ']' {
for {
elem := parse()
elems = append(elems, elem)
b = next()
if b != ',' {
if b != ']' {
fail("got %q, want ',' or ']'", b)
}
break
}
i++ // ','
}
}
i++ // ']'
return starlark.NewList(elems)
case '{':
// object
dict := new(starlark.Dict)
i++ // '{'
b = next()
if b != '}' {
for {
key := parse()
if _, ok := key.(starlark.String); !ok {
fail("got %s for object key, want string", key.Type())
}
b = next()
if b != ':' {
fail("after object key, got %q, want ':' ", b)
}
i++ // ':'
value := parse()
dict.SetKey(key, value) // can't fail
b = next()
if b != ',' {
if b != '}' {
fail("in object, got %q, want ',' or '}'", b)
}
break
}
i++ // ','
}
}
i++ // '}'
return dict
default:
// number?
if isdigit(b) || b == '-' {
// scan literal. Allow [0-9+-eE.] for now.
float := false
var j int
for j = i + 1; j < len(s); j++ {
b = s[j]
if isdigit(b) {
// ok
} else if b == '.' ||
b == 'e' ||
b == 'E' ||
b == '+' ||
b == '-' {
float = true
} else {
break
}
}
num := s[i:j]
i = j
// Unlike most C-like languages,
// JSON disallows a leading zero before a digit.
digits := num
if num[0] == '-' {
digits = num[1:]
}
if digits == "" || digits[0] == '0' && len(digits) > 1 && isdigit(digits[1]) {
fail("invalid number: %s", num)
}
// parse literal
if float {
x, err := strconv.ParseFloat(num, 64)
if err != nil {
fail("invalid number: %s", num)
}
return starlark.Float(x)
} else {
x, ok := new(big.Int).SetString(num, 10)
if !ok {
fail("invalid number: %s", num)
}
return starlark.MakeBigInt(x)
}
}
}
fail("unexpected character %q", b)
panic("unreachable")
}
defer func() {
x := recover()
switch x := x.(type) {
case failure:
if d != nil {
v = d
} else {
err = fmt.Errorf("json.decode: at offset %d, %s", i, x)
}
case nil:
// nop
default:
panic(x) // unexpected panic
}
}()
v = parse()
if skipSpace() {
fail("unexpected character %q after value", s[i])
}
return v, nil
}
func isdigit(b byte) bool {
return b >= '0' && b <= '9'
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/lib/math/math.go
+205
View File
@@ -0,0 +1,205 @@
// Copyright 2021 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package math provides basic constants and mathematical functions.
package math // import "go.starlark.net/lib/math"
import (
"errors"
"fmt"
"math"
"go.starlark.net/starlark"
"go.starlark.net/starlarkstruct"
)
// Module math is a Starlark module of math-related functions and constants.
// The module defines the following functions:
//
// ceil(x) - Returns the ceiling of x, the smallest integer greater than or equal to x.
// copysign(x, y) - Returns a value with the magnitude of x and the sign of y.
// fabs(x) - Returns the absolute value of x as float.
// floor(x) - Returns the floor of x, the largest integer less than or equal to x.
// mod(x, y) - Returns the floating-point remainder of x/y. The magnitude of the result is less than y and its sign agrees with that of x.
// pow(x, y) - Returns x**y, the base-x exponential of y.
// remainder(x, y) - Returns the IEEE 754 floating-point remainder of x/y.
// round(x) - Returns the nearest integer, rounding half away from zero.
//
// exp(x) - Returns e raised to the power x, where e = 2.718281… is the base of natural logarithms.
// sqrt(x) - Returns the square root of x.
//
// acos(x) - Returns the arc cosine of x, in radians.
// asin(x) - Returns the arc sine of x, in radians.
// atan(x) - Returns the arc tangent of x, in radians.
// atan2(y, x) - Returns atan(y / x), in radians.
// The result is between -pi and pi.
// The vector in the plane from the origin to point (x, y) makes this angle with the positive X axis.
// The point of atan2() is that the signs of both inputs are known to it, so it can compute the correct
// quadrant for the angle.
// For example, atan(1) and atan2(1, 1) are both pi/4, but atan2(-1, -1) is -3*pi/4.
// cos(x) - Returns the cosine of x, in radians.
// hypot(x, y) - Returns the Euclidean norm, sqrt(x*x + y*y). This is the length of the vector from the origin to point (x, y).
// sin(x) - Returns the sine of x, in radians.
// tan(x) - Returns the tangent of x, in radians.
//
// degrees(x) - Converts angle x from radians to degrees.
// radians(x) - Converts angle x from degrees to radians.
//
// acosh(x) - Returns the inverse hyperbolic cosine of x.
// asinh(x) - Returns the inverse hyperbolic sine of x.
// atanh(x) - Returns the inverse hyperbolic tangent of x.
// cosh(x) - Returns the hyperbolic cosine of x.
// sinh(x) - Returns the hyperbolic sine of x.
// tanh(x) - Returns the hyperbolic tangent of x.
//
// log(x, base) - Returns the logarithm of x in the given base, or natural logarithm by default.
//
// gamma(x) - Returns the Gamma function of x.
//
// All functions accept both int and float values as arguments.
//
// The module also defines approximations of the following constants:
//
// e - The base of natural logarithms, approximately 2.71828.
// pi - The ratio of a circle's circumference to its diameter, approximately 3.14159.
//
var Module = &starlarkstruct.Module{
Name: "math",
Members: starlark.StringDict{
"ceil": starlark.NewBuiltin("ceil", ceil),
"copysign": newBinaryBuiltin("copysign", math.Copysign),
"fabs": newUnaryBuiltin("fabs", math.Abs),
"floor": starlark.NewBuiltin("floor", floor),
"mod": newBinaryBuiltin("mod", math.Mod),
"pow": newBinaryBuiltin("pow", math.Pow),
"remainder": newBinaryBuiltin("remainder", math.Remainder),
"round": newUnaryBuiltin("round", math.Round),
"exp": newUnaryBuiltin("exp", math.Exp),
"sqrt": newUnaryBuiltin("sqrt", math.Sqrt),
"acos": newUnaryBuiltin("acos", math.Acos),
"asin": newUnaryBuiltin("asin", math.Asin),
"atan": newUnaryBuiltin("atan", math.Atan),
"atan2": newBinaryBuiltin("atan2", math.Atan2),
"cos": newUnaryBuiltin("cos", math.Cos),
"hypot": newBinaryBuiltin("hypot", math.Hypot),
"sin": newUnaryBuiltin("sin", math.Sin),
"tan": newUnaryBuiltin("tan", math.Tan),
"degrees": newUnaryBuiltin("degrees", degrees),
"radians": newUnaryBuiltin("radians", radians),
"acosh": newUnaryBuiltin("acosh", math.Acosh),
"asinh": newUnaryBuiltin("asinh", math.Asinh),
"atanh": newUnaryBuiltin("atanh", math.Atanh),
"cosh": newUnaryBuiltin("cosh", math.Cosh),
"sinh": newUnaryBuiltin("sinh", math.Sinh),
"tanh": newUnaryBuiltin("tanh", math.Tanh),
"log": starlark.NewBuiltin("log", log),
"gamma": newUnaryBuiltin("gamma", math.Gamma),
"e": starlark.Float(math.E),
"pi": starlark.Float(math.Pi),
},
}
// floatOrInt is an Unpacker that converts a Starlark int or float to Go's float64.
type floatOrInt float64
func (p *floatOrInt) Unpack(v starlark.Value) error {
switch v := v.(type) {
case starlark.Int:
*p = floatOrInt(v.Float())
return nil
case starlark.Float:
*p = floatOrInt(v)
return nil
}
return fmt.Errorf("got %s, want float or int", v.Type())
}
// newUnaryBuiltin wraps a unary floating-point Go function
// as a Starlark built-in that accepts int or float arguments.
func newUnaryBuiltin(name string, fn func(float64) float64) *starlark.Builtin {
return starlark.NewBuiltin(name, func(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var x floatOrInt
if err := starlark.UnpackPositionalArgs(name, args, kwargs, 1, &x); err != nil {
return nil, err
}
return starlark.Float(fn(float64(x))), nil
})
}
// newBinaryBuiltin wraps a binary floating-point Go function
// as a Starlark built-in that accepts int or float arguments.
func newBinaryBuiltin(name string, fn func(float64, float64) float64) *starlark.Builtin {
return starlark.NewBuiltin(name, func(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var x, y floatOrInt
if err := starlark.UnpackPositionalArgs(name, args, kwargs, 2, &x, &y); err != nil {
return nil, err
}
return starlark.Float(fn(float64(x), float64(y))), nil
})
}
// log wraps the Log function
// as a Starlark built-in that accepts int or float arguments.
func log(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var (
x floatOrInt
base floatOrInt = math.E
)
if err := starlark.UnpackPositionalArgs("log", args, kwargs, 1, &x, &base); err != nil {
return nil, err
}
if base == 1 {
return nil, errors.New("division by zero")
}
return starlark.Float(math.Log(float64(x)) / math.Log(float64(base))), nil
}
func ceil(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var x starlark.Value
if err := starlark.UnpackPositionalArgs("ceil", args, kwargs, 1, &x); err != nil {
return nil, err
}
switch t := x.(type) {
case starlark.Int:
return t, nil
case starlark.Float:
return starlark.NumberToInt(starlark.Float(math.Ceil(float64(t))))
}
return nil, fmt.Errorf("got %s, want float or int", x.Type())
}
func floor(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var x starlark.Value
if err := starlark.UnpackPositionalArgs("floor", args, kwargs, 1, &x); err != nil {
return nil, err
}
switch t := x.(type) {
case starlark.Int:
return t, nil
case starlark.Float:
return starlark.NumberToInt(starlark.Float(math.Floor(float64(t))))
}
return nil, fmt.Errorf("got %s, want float or int", x.Type())
}
func degrees(x float64) float64 {
return 360 * x / (2 * math.Pi)
}
func radians(x float64) float64 {
return 2 * math.Pi * x / 360
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/lib/proto/cmd/star2proto/star2proto.go
+143
View File
@@ -0,0 +1,143 @@
// The star2proto command executes a Starlark file and prints a protocol
// message, which it expects to find in a module-level variable named 'result'.
//
// THIS COMMAND IS EXPERIMENTAL AND ITS INTERFACE MAY CHANGE.
package main
// TODO(adonovan): add features to make this a useful tool for querying,
// converting, and building messages in proto, JSON, and YAML.
// - define operations for reading and writing files.
// - support (e.g.) querying a proto file given a '-e expr' flag.
// This will need a convenient way to put the relevant descriptors in scope.
import (
"flag"
"fmt"
"log"
"os"
"strings"
"go.starlark.net/lib/json"
starlarkproto "go.starlark.net/lib/proto"
"go.starlark.net/resolve"
"go.starlark.net/starlark"
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/encoding/prototext"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/reflect/protodesc"
"google.golang.org/protobuf/reflect/protoreflect"
"google.golang.org/protobuf/reflect/protoregistry"
"google.golang.org/protobuf/types/descriptorpb"
)
// flags
var (
outputFlag = flag.String("output", "text", "output format (text, wire, json)")
varFlag = flag.String("var", "result", "the variable to output")
descriptors = flag.String("descriptors", "", "comma-separated list of names of files containing proto.FileDescriptorProto messages")
)
// Starlark dialect flags
func init() {
flag.BoolVar(&resolve.AllowSet, "set", resolve.AllowSet, "allow set data type")
// obsolete, no effect:
flag.BoolVar(&resolve.AllowFloat, "fp", true, "allow floating-point numbers")
flag.BoolVar(&resolve.AllowLambda, "lambda", resolve.AllowLambda, "allow lambda expressions")
flag.BoolVar(&resolve.AllowNestedDef, "nesteddef", resolve.AllowNestedDef, "allow nested def statements")
}
func main() {
log.SetPrefix("star2proto: ")
log.SetFlags(0)
flag.Parse()
if len(flag.Args()) != 1 {
fatalf("requires a single Starlark file name")
}
filename := flag.Args()[0]
// By default, use the linked-in descriptors
// (very few in star2proto, e.g. descriptorpb itself).
pool := protoregistry.GlobalFiles
// Load a user-provided FileDescriptorSet produced by a command such as:
// $ protoc --descriptor_set_out=foo.fds foo.proto
if *descriptors != "" {
var fdset descriptorpb.FileDescriptorSet
for i, filename := range strings.Split(*descriptors, ",") {
data, err := os.ReadFile(filename)
if err != nil {
log.Fatalf("--descriptors[%d]: %s", i, err)
}
// Accumulate into the repeated field of FileDescriptors.
if err := (proto.UnmarshalOptions{Merge: true}).Unmarshal(data, &fdset); err != nil {
log.Fatalf("%s does not contain a proto2.FileDescriptorSet: %v", filename, err)
}
}
files, err := protodesc.NewFiles(&fdset)
if err != nil {
log.Fatalf("protodesc.NewFiles: could not build FileDescriptor index: %v", err)
}
pool = files
}
// Execute the Starlark file.
thread := &starlark.Thread{
Print: func(_ *starlark.Thread, msg string) { fmt.Println(msg) },
}
starlarkproto.SetPool(thread, pool)
predeclared := starlark.StringDict{
"proto": starlarkproto.Module,
"json": json.Module,
}
globals, err := starlark.ExecFile(thread, filename, nil, predeclared)
if err != nil {
if evalErr, ok := err.(*starlark.EvalError); ok {
fatalf("%s", evalErr.Backtrace())
} else {
fatalf("%s", err)
}
}
// Print the output variable as a message.
// TODO(adonovan): this is clumsy.
// Let the user call print(), or provide an expression on the command line.
result, ok := globals[*varFlag]
if !ok {
fatalf("%s must define a module-level variable named %q", filename, *varFlag)
}
msgwrap, ok := result.(*starlarkproto.Message)
if !ok {
fatalf("got %s, want proto.Message, for %q", result.Type(), *varFlag)
}
msg := msgwrap.Message()
// -output
var marshal func(protoreflect.ProtoMessage) ([]byte, error)
switch *outputFlag {
case "wire":
marshal = proto.Marshal
case "text":
marshal = prototext.MarshalOptions{Multiline: true, Indent: "\t"}.Marshal
case "json":
marshal = protojson.MarshalOptions{Multiline: true, Indent: "\t"}.Marshal
default:
fatalf("unsupported -output format: %s", *outputFlag)
}
data, err := marshal(msg)
if err != nil {
fatalf("%s", err)
}
os.Stdout.Write(data)
}
func fatalf(format string, args ...interface{}) {
fmt.Fprintf(os.Stderr, "star2proto: ")
fmt.Fprintf(os.Stderr, format, args...)
fmt.Fprintln(os.Stderr)
os.Exit(1)
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/lib/proto/proto.go
+1271
View File
File diff suppressed because it is too large Load Diff
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/lib/time/time.go
+516
View File
@@ -0,0 +1,516 @@
// Copyright 2021 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package time provides time-related constants and functions.
package time // import "go.starlark.net/lib/time"
import (
"errors"
"fmt"
"sort"
"time"
"go.starlark.net/starlark"
"go.starlark.net/starlarkstruct"
"go.starlark.net/syntax"
)
// Module time is a Starlark module of time-related functions and constants.
// The module defines the following functions:
//
// from_timestamp(sec, nsec) - Converts the given Unix time corresponding to the number of seconds
// and (optionally) nanoseconds since January 1, 1970 UTC into an object
// of type Time. For more details, refer to https://pkg.go.dev/time#Unix.
//
// is_valid_timezone(loc) - Reports whether loc is a valid time zone name.
//
// now() - Returns the current local time. Applications may replace this function by a deterministic one.
//
// parse_duration(d) - Parses the given duration string. For more details, refer to
// https://pkg.go.dev/time#ParseDuration.
//
// parse_time(x, format, location) - Parses the given time string using a specific time format and location.
// The expected arguments are a time string (mandatory), a time format
// (optional, set to RFC3339 by default, e.g. "2021-03-22T23:20:50.52Z")
// and a name of location (optional, set to UTC by default). For more details,
// refer to https://pkg.go.dev/time#Parse and https://pkg.go.dev/time#ParseInLocation.
//
// time(year, month, day, hour, minute, second, nanosecond, location) - Returns the Time corresponding to
// yyyy-mm-dd hh:mm:ss + nsec nanoseconds
// in the appropriate zone for that time
// in the given location. All the parameters
// are optional.
// The module also defines the following constants:
//
// nanosecond - A duration representing one nanosecond.
// microsecond - A duration representing one microsecond.
// millisecond - A duration representing one millisecond.
// second - A duration representing one second.
// minute - A duration representing one minute.
// hour - A duration representing one hour.
//
var Module = &starlarkstruct.Module{
Name: "time",
Members: starlark.StringDict{
"from_timestamp": starlark.NewBuiltin("from_timestamp", fromTimestamp),
"is_valid_timezone": starlark.NewBuiltin("is_valid_timezone", isValidTimezone),
"now": starlark.NewBuiltin("now", now),
"parse_duration": starlark.NewBuiltin("parse_duration", parseDuration),
"parse_time": starlark.NewBuiltin("parse_time", parseTime),
"time": starlark.NewBuiltin("time", newTime),
"nanosecond": Duration(time.Nanosecond),
"microsecond": Duration(time.Microsecond),
"millisecond": Duration(time.Millisecond),
"second": Duration(time.Second),
"minute": Duration(time.Minute),
"hour": Duration(time.Hour),
},
}
// NowFunc is a function that reports the current time. Intentionally exported
// so that it can be overridden, for example by applications that require their
// Starlark scripts to be fully deterministic.
//
// Deprecated: avoid updating this global variable
// and instead use SetNow on each thread to set its clock function.
var NowFunc = time.Now
const contextKey = "time.now"
// SetNow sets the thread's optional clock function.
// If non-nil, it will be used in preference to NowFunc when the
// thread requests the current time by executing a call to time.now.
func SetNow(thread *starlark.Thread, nowFunc func() (time.Time, error)) {
thread.SetLocal(contextKey, nowFunc)
}
// Now returns the clock function previously associated with this thread.
func Now(thread *starlark.Thread) func() (time.Time, error) {
nowFunc, _ := thread.Local(contextKey).(func() (time.Time, error))
return nowFunc
}
func parseDuration(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var d Duration
err := starlark.UnpackPositionalArgs("parse_duration", args, kwargs, 1, &d)
return d, err
}
func isValidTimezone(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var s string
if err := starlark.UnpackPositionalArgs("is_valid_timezone", args, kwargs, 1, &s); err != nil {
return nil, err
}
_, err := time.LoadLocation(s)
return starlark.Bool(err == nil), nil
}
func parseTime(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var (
x string
location = "UTC"
format = time.RFC3339
)
if err := starlark.UnpackArgs("parse_time", args, kwargs, "x", &x, "format?", &format, "location?", &location); err != nil {
return nil, err
}
if location == "UTC" {
t, err := time.Parse(format, x)
if err != nil {
return nil, err
}
return Time(t), nil
}
loc, err := time.LoadLocation(location)
if err != nil {
return nil, err
}
t, err := time.ParseInLocation(format, x, loc)
if err != nil {
return nil, err
}
return Time(t), nil
}
func fromTimestamp(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var (
sec int64
nsec int64 = 0
)
if err := starlark.UnpackPositionalArgs("from_timestamp", args, kwargs, 1, &sec, &nsec); err != nil {
return nil, err
}
return Time(time.Unix(sec, nsec)), nil
}
func now(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
nowErrFunc := Now(thread)
if nowErrFunc != nil {
t, err := nowErrFunc()
if err != nil {
return nil, err
}
return Time(t), nil
}
nowFunc := NowFunc
if nowFunc == nil {
return nil, errors.New("time.now() is not available")
}
return Time(nowFunc()), nil
}
// Duration is a Starlark representation of a duration.
type Duration time.Duration
// Assert at compile time that Duration implements Unpacker.
var _ starlark.Unpacker = (*Duration)(nil)
// Unpack is a custom argument unpacker
func (d *Duration) Unpack(v starlark.Value) error {
switch x := v.(type) {
case Duration:
*d = x
return nil
case starlark.String:
dur, err := time.ParseDuration(string(x))
if err != nil {
return err
}
*d = Duration(dur)
return nil
}
return fmt.Errorf("got %s, want a duration, string, or int", v.Type())
}
// String implements the Stringer interface.
func (d Duration) String() string { return time.Duration(d).String() }
// Type returns a short string describing the value's type.
func (d Duration) Type() string { return "time.duration" }
// Freeze renders Duration immutable. required by starlark.Value interface
// because duration is already immutable this is a no-op.
func (d Duration) Freeze() {}
// Hash returns a function of x such that Equals(x, y) => Hash(x) == Hash(y)
// required by starlark.Value interface.
func (d Duration) Hash() (uint32, error) {
return uint32(d) ^ uint32(int64(d)>>32), nil
}
// Truth reports whether the duration is non-zero.
func (d Duration) Truth() starlark.Bool { return d != 0 }
// Attr gets a value for a string attribute, implementing dot expression support
// in starklark. required by starlark.HasAttrs interface.
func (d Duration) Attr(name string) (starlark.Value, error) {
switch name {
case "hours":
return starlark.Float(time.Duration(d).Hours()), nil
case "minutes":
return starlark.Float(time.Duration(d).Minutes()), nil
case "seconds":
return starlark.Float(time.Duration(d).Seconds()), nil
case "milliseconds":
return starlark.MakeInt64(time.Duration(d).Milliseconds()), nil
case "microseconds":
return starlark.MakeInt64(time.Duration(d).Microseconds()), nil
case "nanoseconds":
return starlark.MakeInt64(time.Duration(d).Nanoseconds()), nil
}
return nil, fmt.Errorf("unrecognized %s attribute %q", d.Type(), name)
}
// AttrNames lists available dot expression strings. required by
// starlark.HasAttrs interface.
func (d Duration) AttrNames() []string {
return []string{
"hours",
"minutes",
"seconds",
"milliseconds",
"microseconds",
"nanoseconds",
}
}
// Cmp implements comparison of two Duration values. required by
// starlark.TotallyOrdered interface.
func (d Duration) Cmp(v starlark.Value, depth int) (int, error) {
if x, y := d, v.(Duration); x < y {
return -1, nil
} else if x > y {
return 1, nil
}
return 0, nil
}
// Binary implements binary operators, which satisfies the starlark.HasBinary
// interface. operators:
// duration + duration = duration
// duration + time = time
// duration - duration = duration
// duration / duration = float
// duration / int = duration
// duration / float = duration
// duration // duration = int
// duration * int = duration
func (d Duration) Binary(op syntax.Token, y starlark.Value, side starlark.Side) (starlark.Value, error) {
x := time.Duration(d)
switch op {
case syntax.PLUS:
switch y := y.(type) {
case Duration:
return Duration(x + time.Duration(y)), nil
case Time:
return Time(time.Time(y).Add(x)), nil
}
case syntax.MINUS:
switch y := y.(type) {
case Duration:
return Duration(x - time.Duration(y)), nil
}
case syntax.SLASH:
switch y := y.(type) {
case Duration:
if y == 0 {
return nil, fmt.Errorf("%s division by zero", d.Type())
}
return starlark.Float(x.Nanoseconds()) / starlark.Float(time.Duration(y).Nanoseconds()), nil
case starlark.Int:
if side == starlark.Right {
return nil, fmt.Errorf("unsupported operation")
}
i, ok := y.Int64()
if !ok {
return nil, fmt.Errorf("int value out of range (want signed 64-bit value)")
}
if i == 0 {
return nil, fmt.Errorf("%s division by zero", d.Type())
}
return d / Duration(i), nil
case starlark.Float:
f := float64(y)
if f == 0 {
return nil, fmt.Errorf("%s division by zero", d.Type())
}
return Duration(float64(x.Nanoseconds()) / f), nil
}
case syntax.SLASHSLASH:
switch y := y.(type) {
case Duration:
if y == 0 {
return nil, fmt.Errorf("%s division by zero", d.Type())
}
return starlark.MakeInt64(x.Nanoseconds() / time.Duration(y).Nanoseconds()), nil
}
case syntax.STAR:
switch y := y.(type) {
case starlark.Int:
i, ok := y.Int64()
if !ok {
return nil, fmt.Errorf("int value out of range (want signed 64-bit value)")
}
return d * Duration(i), nil
}
}
return nil, nil
}
// Time is a Starlark representation of a moment in time.
type Time time.Time
func newTime(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var (
year, month, day, hour, min, sec, nsec int
loc string
)
if err := starlark.UnpackArgs("time", args, kwargs,
"year?", &year,
"month?", &month,
"day?", &day,
"hour?", &hour,
"minute?", &min,
"second?", &sec,
"nanosecond?", &nsec,
"location?", &loc,
); err != nil {
return nil, err
}
if len(args) > 0 {
return nil, fmt.Errorf("time: unexpected positional arguments")
}
location, err := time.LoadLocation(loc)
if err != nil {
return nil, err
}
return Time(time.Date(year, time.Month(month), day, hour, min, sec, nsec, location)), nil
}
// String returns the time formatted using the format string
// "2006-01-02 15:04:05.999999999 -0700 MST".
func (t Time) String() string { return time.Time(t).String() }
// Type returns "time.time".
func (t Time) Type() string { return "time.time" }
// Freeze renders time immutable. required by starlark.Value interface
// because Time is already immutable this is a no-op.
func (t Time) Freeze() {}
// Hash returns a function of x such that Equals(x, y) => Hash(x) == Hash(y)
// required by starlark.Value interface.
func (t Time) Hash() (uint32, error) {
return uint32(time.Time(t).UnixNano()) ^ uint32(int64(time.Time(t).UnixNano())>>32), nil
}
// Truth returns the truth value of an object required by starlark.Value
// interface.
func (t Time) Truth() starlark.Bool { return !starlark.Bool(time.Time(t).IsZero()) }
// Attr gets a value for a string attribute, implementing dot expression support
// in starklark. required by starlark.HasAttrs interface.
func (t Time) Attr(name string) (starlark.Value, error) {
switch name {
case "year":
return starlark.MakeInt(time.Time(t).Year()), nil
case "month":
return starlark.MakeInt(int(time.Time(t).Month())), nil
case "day":
return starlark.MakeInt(time.Time(t).Day()), nil
case "hour":
return starlark.MakeInt(time.Time(t).Hour()), nil
case "minute":
return starlark.MakeInt(time.Time(t).Minute()), nil
case "second":
return starlark.MakeInt(time.Time(t).Second()), nil
case "nanosecond":
return starlark.MakeInt(time.Time(t).Nanosecond()), nil
case "unix":
return starlark.MakeInt64(time.Time(t).Unix()), nil
case "unix_nano":
return starlark.MakeInt64(time.Time(t).UnixNano()), nil
}
return builtinAttr(t, name, timeMethods)
}
// AttrNames lists available dot expression strings for time. required by
// starlark.HasAttrs interface.
func (t Time) AttrNames() []string {
return append(builtinAttrNames(timeMethods),
"year",
"month",
"day",
"hour",
"minute",
"second",
"nanosecond",
"unix",
"unix_nano",
)
}
// Cmp implements comparison of two Time values. Required by
// starlark.TotallyOrdered interface.
func (t Time) Cmp(yV starlark.Value, depth int) (int, error) {
x := time.Time(t)
y := time.Time(yV.(Time))
if x.Before(y) {
return -1, nil
} else if x.After(y) {
return 1, nil
}
return 0, nil
}
// Binary implements binary operators, which satisfies the starlark.HasBinary
// interface
// time + duration = time
// time - duration = time
// time - time = duration
func (t Time) Binary(op syntax.Token, y starlark.Value, side starlark.Side) (starlark.Value, error) {
x := time.Time(t)
switch op {
case syntax.PLUS:
switch y := y.(type) {
case Duration:
return Time(x.Add(time.Duration(y))), nil
}
case syntax.MINUS:
switch y := y.(type) {
case Duration:
return Time(x.Add(time.Duration(-y))), nil
case Time:
// time - time = duration
return Duration(x.Sub(time.Time(y))), nil
}
}
return nil, nil
}
var timeMethods = map[string]builtinMethod{
"in_location": timeIn,
"format": timeFormat,
}
func timeFormat(fnname string, recV starlark.Value, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var x string
if err := starlark.UnpackPositionalArgs("format", args, kwargs, 1, &x); err != nil {
return nil, err
}
recv := time.Time(recV.(Time))
return starlark.String(recv.Format(x)), nil
}
func timeIn(fnname string, recV starlark.Value, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var x string
if err := starlark.UnpackPositionalArgs("in_location", args, kwargs, 1, &x); err != nil {
return nil, err
}
loc, err := time.LoadLocation(x)
if err != nil {
return nil, err
}
recv := time.Time(recV.(Time))
return Time(recv.In(loc)), nil
}
type builtinMethod func(fnname string, recv starlark.Value, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error)
func builtinAttr(recv starlark.Value, name string, methods map[string]builtinMethod) (starlark.Value, error) {
method := methods[name]
if method == nil {
return nil, nil // no such method
}
// Allocate a closure over 'method'.
impl := func(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
return method(b.Name(), b.Receiver(), args, kwargs)
}
return starlark.NewBuiltin(name, impl).BindReceiver(recv), nil
}
func builtinAttrNames(methods map[string]builtinMethod) []string {
names := make([]string, 0, len(methods))
for name := range methods {
names = append(names, name)
}
sort.Strings(names)
return names
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/lib/time/time_test.go
+82
View File
@@ -0,0 +1,82 @@
package time
import (
"errors"
"testing"
"time"
"go.starlark.net/starlark"
)
func TestPerThreadNowReturnsCorrectTime(t *testing.T) {
th := &starlark.Thread{}
date := time.Date(1, 2, 3, 4, 5, 6, 7, time.UTC)
SetNow(th, func() (time.Time, error) {
return date, nil
})
res, err := starlark.Call(th, Module.Members["now"], nil, nil)
if err != nil {
t.Fatal(err)
}
retTime := time.Time(res.(Time))
if !retTime.Equal(date) {
t.Fatal("Expected time to be equal", retTime, date)
}
}
func TestPerThreadNowReturnsError(t *testing.T) {
th := &starlark.Thread{}
e := errors.New("no time")
SetNow(th, func() (time.Time, error) {
return time.Time{}, e
})
_, err := starlark.Call(th, Module.Members["now"], nil, nil)
if !errors.Is(err, e) {
t.Fatal("Expected equal error", e, err)
}
}
func TestGlobalNowReturnsCorrectTime(t *testing.T) {
th := &starlark.Thread{}
oldNow := NowFunc
defer func() {
NowFunc = oldNow
}()
date := time.Date(1, 2, 3, 4, 5, 6, 7, time.UTC)
NowFunc = func() time.Time {
return date
}
res, err := starlark.Call(th, Module.Members["now"], nil, nil)
if err != nil {
t.Fatal(err)
}
retTime := time.Time(res.(Time))
if !retTime.Equal(date) {
t.Fatal("Expected time to be equal", retTime, date)
}
}
func TestGlobalNowReturnsErrorWhenNil(t *testing.T) {
th := &starlark.Thread{}
oldNow := NowFunc
defer func() {
NowFunc = oldNow
}()
NowFunc = nil
_, err := starlark.Call(th, Module.Members["now"], nil, nil)
if err == nil {
t.Fatal("Expected to get an error")
}
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/repl/repl.go
+193
View File
@@ -0,0 +1,193 @@
// Package repl provides a read/eval/print loop for Starlark.
//
// It supports readline-style command editing,
// and interrupts through Control-C.
//
// If an input line can be parsed as an expression,
// the REPL parses and evaluates it and prints its result.
// Otherwise the REPL reads lines until a blank line,
// then tries again to parse the multi-line input as an
// expression. If the input still cannot be parsed as an expression,
// the REPL parses and executes it as a file (a list of statements),
// for side effects.
package repl // import "go.starlark.net/repl"
import (
"context"
"fmt"
"io"
"os"
"os/signal"
"github.com/chzyer/readline"
"go.starlark.net/starlark"
"go.starlark.net/syntax"
)
var interrupted = make(chan os.Signal, 1)
// REPL calls [REPLOptions] using [syntax.LegacyFileOptions].
// Deprecated: relies on legacy global variables.
func REPL(thread *starlark.Thread, globals starlark.StringDict) {
REPLOptions(syntax.LegacyFileOptions(), thread, globals)
}
// REPLOptions executes a read, eval, print loop.
//
// Before evaluating each expression, it sets the Starlark thread local
// variable named "context" to a context.Context that is cancelled by a
// SIGINT (Control-C). Client-supplied global functions may use this
// context to make long-running operations interruptable.
func REPLOptions(opts *syntax.FileOptions, thread *starlark.Thread, globals starlark.StringDict) {
signal.Notify(interrupted, os.Interrupt)
defer signal.Stop(interrupted)
rl, err := readline.New(">>> ")
if err != nil {
PrintError(err)
return
}
defer rl.Close()
for {
if err := rep(opts, rl, thread, globals); err != nil {
if err == readline.ErrInterrupt {
fmt.Println(err)
continue
}
break
}
}
}
// rep reads, evaluates, and prints one item.
//
// It returns an error (possibly readline.ErrInterrupt)
// only if readline failed. Starlark errors are printed.
func rep(opts *syntax.FileOptions, rl *readline.Instance, thread *starlark.Thread, globals starlark.StringDict) error {
// Each item gets its own context,
// which is cancelled by a SIGINT.
//
// Note: during Readline calls, Control-C causes Readline to return
// ErrInterrupt but does not generate a SIGINT.
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go func() {
select {
case <-interrupted:
cancel()
case <-ctx.Done():
}
}()
thread.SetLocal("context", ctx)
eof := false
// readline returns EOF, ErrInterrupted, or a line including "\n".
rl.SetPrompt(">>> ")
readline := func() ([]byte, error) {
line, err := rl.Readline()
rl.SetPrompt("... ")
if err != nil {
if err == io.EOF {
eof = true
}
return nil, err
}
return []byte(line + "\n"), nil
}
// Treat load bindings as global (like they used to be) in the REPL.
// Fixes github.com/google/starlark-go/issues/224.
opts2 := *opts
opts2.LoadBindsGlobally = true
opts = &opts2
// parse
f, err := opts.ParseCompoundStmt("<stdin>", readline)
if err != nil {
if eof {
return io.EOF
}
PrintError(err)
return nil
}
if expr := soleExpr(f); expr != nil {
// eval
v, err := starlark.EvalExprOptions(f.Options, thread, expr, globals)
if err != nil {
PrintError(err)
return nil
}
// print
if v != starlark.None {
fmt.Println(v)
}
} else if err := starlark.ExecREPLChunk(f, thread, globals); err != nil {
PrintError(err)
return nil
}
return nil
}
func soleExpr(f *syntax.File) syntax.Expr {
if len(f.Stmts) == 1 {
if stmt, ok := f.Stmts[0].(*syntax.ExprStmt); ok {
return stmt.X
}
}
return nil
}
// PrintError prints the error to stderr,
// or its backtrace if it is a Starlark evaluation error.
func PrintError(err error) {
if evalErr, ok := err.(*starlark.EvalError); ok {
fmt.Fprintln(os.Stderr, evalErr.Backtrace())
} else {
fmt.Fprintln(os.Stderr, err)
}
}
// MakeLoad calls [MakeLoadOptions] using [syntax.LegacyFileOptions].
// Deprecated: relies on legacy global variables.
func MakeLoad() func(thread *starlark.Thread, module string) (starlark.StringDict, error) {
return MakeLoadOptions(syntax.LegacyFileOptions())
}
// MakeLoadOptions returns a simple sequential implementation of module loading
// suitable for use in the REPL.
// Each function returned by MakeLoadOptions accesses a distinct private cache.
func MakeLoadOptions(opts *syntax.FileOptions) func(thread *starlark.Thread, module string) (starlark.StringDict, error) {
type entry struct {
globals starlark.StringDict
err error
}
var cache = make(map[string]*entry)
return func(thread *starlark.Thread, module string) (starlark.StringDict, error) {
e, ok := cache[module]
if e == nil {
if ok {
// request for package whose loading is in progress
return nil, fmt.Errorf("cycle in load graph")
}
// Add a placeholder to indicate "load in progress".
cache[module] = nil
// Load it.
thread := &starlark.Thread{Name: "exec " + module, Load: thread.Load}
globals, err := starlark.ExecFileOptions(opts, thread, module, nil, nil)
e = &entry{globals, err}
// Update the cache.
cache[module] = e
}
return e.globals, e.err
}
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/resolve/binding.go
+74
View File
@@ -0,0 +1,74 @@
// Copyright 2019 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package resolve
import "go.starlark.net/syntax"
// This file defines resolver data types saved in the syntax tree.
// We cannot guarantee API stability for these types
// as they are closely tied to the implementation.
// A Binding contains resolver information about an identifier.
// The resolver populates the Binding field of each syntax.Identifier.
// The Binding ties together all identifiers that denote the same variable.
type Binding struct {
Scope Scope
// Index records the index into the enclosing
// - {DefStmt,File}.Locals, if Scope==Local
// - DefStmt.FreeVars, if Scope==Free
// - File.Globals, if Scope==Global.
// It is zero if Scope is Predeclared, Universal, or Undefined.
Index int
First *syntax.Ident // first binding use (iff Scope==Local/Free/Global)
}
// The Scope of Binding indicates what kind of scope it has.
type Scope uint8
const (
Undefined Scope = iota // name is not defined
Local // name is local to its function or file
Cell // name is function-local but shared with a nested function
Free // name is cell of some enclosing function
Global // name is global to module
Predeclared // name is predeclared for this module (e.g. glob)
Universal // name is universal (e.g. len)
)
var scopeNames = [...]string{
Undefined: "undefined",
Local: "local",
Cell: "cell",
Free: "free",
Global: "global",
Predeclared: "predeclared",
Universal: "universal",
}
func (scope Scope) String() string { return scopeNames[scope] }
// A Module contains resolver information about a file.
// The resolver populates the Module field of each syntax.File.
type Module struct {
Locals []*Binding // the file's (comprehension-)local variables
Globals []*Binding // the file's global variables
}
// A Function contains resolver information about a named or anonymous function.
// The resolver populates the Function field of each syntax.DefStmt and syntax.LambdaExpr.
type Function struct {
Pos syntax.Position // of DEF or LAMBDA
Name string // name of def, or "lambda"
Params []syntax.Expr // param = ident | ident=expr | * | *ident | **ident
Body []syntax.Stmt // contains synthetic 'return expr' for lambda
HasVarargs bool // whether params includes *args (convenience)
HasKwargs bool // whether params includes **kwargs (convenience)
NumKwonlyParams int // number of keyword-only optional parameters
Locals []*Binding // this function's local/cell variables, parameters first
FreeVars []*Binding // enclosing cells to capture in closure
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/resolve/resolve.go
+981
View File
@@ -0,0 +1,981 @@
// Copyright 2017 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package resolve defines a name-resolution pass for Starlark abstract
// syntax trees.
//
// The resolver sets the Locals and FreeVars arrays of each DefStmt and
// the LocalIndex field of each syntax.Ident that refers to a local or
// free variable. It also sets the Locals array of a File for locals
// bound by top-level comprehensions and load statements.
// Identifiers for global variables do not get an index.
package resolve // import "go.starlark.net/resolve"
// All references to names are statically resolved. Names may be
// predeclared, global, or local to a function or file.
// File-local variables include those bound by top-level comprehensions
// and by load statements. ("Top-level" means "outside of any function".)
// The resolver maps each global name to a small integer and each local
// name to a small integer; these integers enable a fast and compact
// representation of globals and locals in the evaluator.
//
// As an optimization, the resolver classifies each predeclared name as
// either universal (e.g. None, len) or per-module (e.g. glob in Bazel's
// build language), enabling the evaluator to share the representation
// of the universal environment across all modules.
//
// The lexical environment is a tree of blocks with the file block at
// its root. The file's child blocks may be of two kinds: functions
// and comprehensions, and these may have further children of either
// kind.
//
// Python-style resolution requires multiple passes because a name is
// determined to be local to a function only if the function contains a
// "binding" use of it; similarly, a name is determined to be global (as
// opposed to predeclared) if the module contains a top-level binding use.
// Unlike ordinary top-level assignments, the bindings created by load
// statements are local to the file block.
// A non-binding use may lexically precede the binding to which it is resolved.
// In the first pass, we inspect each function, recording in
// 'uses' each identifier and the environment block in which it occurs.
// If a use of a name is binding, such as a function parameter or
// assignment, we add the name to the block's bindings mapping and add a
// local variable to the enclosing function.
//
// As we finish resolving each function, we inspect all the uses within
// that function and discard ones that were found to be function-local. The
// remaining ones must be either free (local to some lexically enclosing
// function), or top-level (global, predeclared, or file-local), but we cannot tell
// which until we have finished inspecting the outermost enclosing
// function. At that point, we can distinguish local from top-level names
// (and this is when Python would compute free variables).
//
// However, Starlark additionally requires that all references to global
// names are satisfied by some declaration in the current module;
// Starlark permits a function to forward-reference a global or file-local
// that has not
// been declared yet so long as it is declared before the end of the
// module. So, instead of re-resolving the unresolved references after
// each top-level function, we defer this until the end of the module
// and ensure that all such references are satisfied by some definition.
//
// At the end of the module, we visit each of the nested function blocks
// in bottom-up order, doing a recursive lexical lookup for each
// unresolved name. If the name is found to be local to some enclosing
// function, we must create a DefStmt.FreeVar (capture) parameter for
// each intervening function. We enter these synthetic bindings into
// the bindings map so that we create at most one freevar per name. If
// the name was not local, we check that it was defined at module level.
//
// We resolve all uses of locals in the module (due to load statements
// and comprehensions) in a similar way and compute the file's set of
// local variables.
//
// Starlark enforces that all global names are assigned at most once on
// all control flow paths by forbidding if/else statements and loops at
// top level. A global may be used before it is defined, leading to a
// dynamic error. However, the AllowGlobalReassign flag (really: allow
// top-level reassign) makes the resolver allow multiple to a variable
// at top-level. It also allows if-, for-, and while-loops at top-level,
// which in turn may make the evaluator dynamically assign multiple
// values to a variable at top-level. (These two roles should be separated.)
import (
"fmt"
"log"
"sort"
"strings"
"go.starlark.net/internal/spell"
"go.starlark.net/syntax"
)
const debug = false
const doesnt = "this Starlark dialect does not "
// global options
// These features are either not standard Starlark (yet), or deprecated
// features of the BUILD language, so we put them behind flags.
//
// Deprecated: use an explicit [syntax.FileOptions] argument instead,
// as it avoids all the usual problems of global variables.
var (
AllowSet = false // allow the 'set' built-in
AllowGlobalReassign = false // allow reassignment to top-level names; also, allow if/for/while at top-level
AllowRecursion = false // allow while statements and recursive functions
LoadBindsGlobally = false // load creates global not file-local bindings (deprecated)
// obsolete flags for features that are now standard. No effect.
AllowNestedDef = true
AllowLambda = true
AllowFloat = true
AllowBitwise = true
)
// File resolves the specified file and records information about the
// module in file.Module.
//
// The isPredeclared and isUniversal predicates report whether a name is
// a pre-declared identifier (visible in the current module) or a
// universal identifier (visible in every module).
// Clients should typically pass predeclared.Has for the first and
// starlark.Universe.Has for the second, where predeclared is the
// module's StringDict of predeclared names and starlark.Universe is the
// standard set of built-ins.
// The isUniverse predicate is supplied a parameter to avoid a cyclic
// dependency upon starlark.Universe, not because users should ever need
// to redefine it.
func File(file *syntax.File, isPredeclared, isUniversal func(name string) bool) error {
return REPLChunk(file, nil, isPredeclared, isUniversal)
}
// REPLChunk is a generalization of the File function that supports a
// non-empty initial global block, as occurs in a REPL.
func REPLChunk(file *syntax.File, isGlobal, isPredeclared, isUniversal func(name string) bool) error {
r := newResolver(file.Options, isGlobal, isPredeclared, isUniversal)
r.stmts(file.Stmts)
r.env.resolveLocalUses()
// At the end of the module, resolve all non-local variable references,
// computing closures.
// Function bodies may contain forward references to later global declarations.
r.resolveNonLocalUses(r.env)
file.Module = &Module{
Locals: r.moduleLocals,
Globals: r.moduleGlobals,
}
if len(r.errors) > 0 {
return r.errors
}
return nil
}
// Expr calls [ExprOptions] using [syntax.LegacyFileOptions].
// Deprecated: relies on legacy global variables.
func Expr(expr syntax.Expr, isPredeclared, isUniversal func(name string) bool) ([]*Binding, error) {
return ExprOptions(syntax.LegacyFileOptions(), expr, isPredeclared, isUniversal)
}
// ExprOptions resolves the specified expression.
// It returns the local variables bound within the expression.
//
// The isPredeclared and isUniversal predicates behave as for the File function
func ExprOptions(opts *syntax.FileOptions, expr syntax.Expr, isPredeclared, isUniversal func(name string) bool) ([]*Binding, error) {
r := newResolver(opts, nil, isPredeclared, isUniversal)
r.expr(expr)
r.env.resolveLocalUses()
r.resolveNonLocalUses(r.env) // globals & universals
if len(r.errors) > 0 {
return nil, r.errors
}
return r.moduleLocals, nil
}
// An ErrorList is a non-empty list of resolver error messages.
type ErrorList []Error // len > 0
func (e ErrorList) Error() string { return e[0].Error() }
// An Error describes the nature and position of a resolver error.
type Error struct {
Pos syntax.Position
Msg string
}
func (e Error) Error() string { return e.Pos.String() + ": " + e.Msg }
func newResolver(options *syntax.FileOptions, isGlobal, isPredeclared, isUniversal func(name string) bool) *resolver {
file := new(block)
return &resolver{
options: options,
file: file,
env: file,
isGlobal: isGlobal,
isPredeclared: isPredeclared,
isUniversal: isUniversal,
globals: make(map[string]*Binding),
predeclared: make(map[string]*Binding),
}
}
type resolver struct {
options *syntax.FileOptions
// env is the current local environment:
// a linked list of blocks, innermost first.
// The tail of the list is the file block.
env *block
file *block // file block (contains load bindings)
// moduleLocals contains the local variables of the module
// (due to load statements and comprehensions outside any function).
// moduleGlobals contains the global variables of the module.
moduleLocals []*Binding
moduleGlobals []*Binding
// globals maps each global name in the module to its binding.
// predeclared does the same for predeclared and universal names.
globals map[string]*Binding
predeclared map[string]*Binding
// These predicates report whether a name is
// pre-declared, either in this module or universally,
// or already declared in the module globals (as in a REPL).
// isGlobal may be nil.
isGlobal, isPredeclared, isUniversal func(name string) bool
loops int // number of enclosing for/while loops
ifstmts int // number of enclosing if statements loops
errors ErrorList
}
// container returns the innermost enclosing "container" block:
// a function (function != nil) or file (function == nil).
// Container blocks accumulate local variable bindings.
func (r *resolver) container() *block {
for b := r.env; ; b = b.parent {
if b.function != nil || b == r.file {
return b
}
}
}
func (r *resolver) push(b *block) {
r.env.children = append(r.env.children, b)
b.parent = r.env
r.env = b
}
func (r *resolver) pop() { r.env = r.env.parent }
type block struct {
parent *block // nil for file block
// In the file (root) block, both these fields are nil.
function *Function // only for function blocks
comp *syntax.Comprehension // only for comprehension blocks
// bindings maps a name to its binding.
// A local binding has an index into its innermost enclosing container's locals array.
// A free binding has an index into its innermost enclosing function's freevars array.
bindings map[string]*Binding
// children records the child blocks of the current one.
children []*block
// uses records all identifiers seen in this container (function or file),
// and a reference to the environment in which they appear.
// As we leave each container block, we resolve them,
// so that only free and global ones remain.
// At the end of each top-level function we compute closures.
uses []use
}
func (b *block) bind(name string, bind *Binding) {
if b.bindings == nil {
b.bindings = make(map[string]*Binding)
}
b.bindings[name] = bind
}
func (b *block) String() string {
if b.function != nil {
return "function block at " + fmt.Sprint(b.function.Pos)
}
if b.comp != nil {
return "comprehension block at " + fmt.Sprint(b.comp.Span())
}
return "file block"
}
func (r *resolver) errorf(posn syntax.Position, format string, args ...interface{}) {
r.errors = append(r.errors, Error{posn, fmt.Sprintf(format, args...)})
}
// A use records an identifier and the environment in which it appears.
type use struct {
id *syntax.Ident
env *block
}
// bind creates a binding for id: a global (not file-local)
// binding at top-level, a local binding otherwise.
// At top-level, it reports an error if a global or file-local
// binding already exists, unless AllowGlobalReassign.
// It sets id.Binding to the binding (whether old or new),
// and returns whether a binding already existed.
func (r *resolver) bind(id *syntax.Ident) bool {
// Binding outside any local (comprehension/function) block?
if r.env == r.file {
bind, ok := r.file.bindings[id.Name]
if !ok {
bind, ok = r.globals[id.Name]
if !ok {
// first global binding of this name
bind = &Binding{
First: id,
Scope: Global,
Index: len(r.moduleGlobals),
}
r.globals[id.Name] = bind
r.moduleGlobals = append(r.moduleGlobals, bind)
}
}
if ok && !r.options.GlobalReassign {
r.errorf(id.NamePos, "cannot reassign %s %s declared at %s",
bind.Scope, id.Name, bind.First.NamePos)
}
id.Binding = bind
return ok
}
return r.bindLocal(id)
}
func (r *resolver) bindLocal(id *syntax.Ident) bool {
// Mark this name as local to current block.
// Assign it a new local (positive) index in the current container.
_, ok := r.env.bindings[id.Name]
if !ok {
var locals *[]*Binding
if fn := r.container().function; fn != nil {
locals = &fn.Locals
} else {
locals = &r.moduleLocals
}
bind := &Binding{
First: id,
Scope: Local,
Index: len(*locals),
}
r.env.bind(id.Name, bind)
*locals = append(*locals, bind)
}
r.use(id)
return ok
}
func (r *resolver) use(id *syntax.Ident) {
use := use{id, r.env}
// The spec says that if there is a global binding of a name
// then all references to that name in that block refer to the
// global, even if the use precedes the def---just as for locals.
// For example, in this code,
//
// print(len); len=1; print(len)
//
// both occurrences of len refer to the len=1 binding, which
// completely shadows the predeclared len function.
//
// The rationale for these semantics, which differ from Python,
// is that the static meaning of len (a reference to a global)
// does not change depending on where it appears in the file.
// Of course, its dynamic meaning does change, from an error
// into a valid reference, so it's not clear these semantics
// have any practical advantage.
//
// In any case, the Bazel implementation lags behind the spec
// and follows Python behavior, so the first use of len refers
// to the predeclared function. This typically used in a BUILD
// file that redefines a predeclared name half way through,
// for example:
//
// proto_library(...) # built-in rule
// load("myproto.bzl", "proto_library")
// proto_library(...) # user-defined rule
//
// We will piggyback support for the legacy semantics on the
// AllowGlobalReassign flag, which is loosely related and also
// required for Bazel.
if r.options.GlobalReassign && r.env == r.file {
r.useToplevel(use)
return
}
b := r.container()
b.uses = append(b.uses, use)
}
// useToplevel resolves use.id as a reference to a name visible at top-level.
// The use.env field captures the original environment for error reporting.
func (r *resolver) useToplevel(use use) (bind *Binding) {
id := use.id
if prev, ok := r.file.bindings[id.Name]; ok {
// use of load-defined name in file block
bind = prev
} else if prev, ok := r.globals[id.Name]; ok {
// use of global declared by module
bind = prev
} else if r.isGlobal != nil && r.isGlobal(id.Name) {
// use of global defined in a previous REPL chunk
bind = &Binding{
First: id, // wrong: this is not even a binding use
Scope: Global,
Index: len(r.moduleGlobals),
}
r.globals[id.Name] = bind
r.moduleGlobals = append(r.moduleGlobals, bind)
} else if prev, ok := r.predeclared[id.Name]; ok {
// repeated use of predeclared or universal
bind = prev
} else if r.isPredeclared(id.Name) {
// use of pre-declared name
bind = &Binding{Scope: Predeclared}
r.predeclared[id.Name] = bind // save it
} else if r.isUniversal(id.Name) {
// use of universal name
if !r.options.Set && id.Name == "set" {
r.errorf(id.NamePos, doesnt+"support sets")
}
bind = &Binding{Scope: Universal}
r.predeclared[id.Name] = bind // save it
} else {
bind = &Binding{Scope: Undefined}
var hint string
if n := r.spellcheck(use); n != "" {
hint = fmt.Sprintf(" (did you mean %s?)", n)
}
r.errorf(id.NamePos, "undefined: %s%s", id.Name, hint)
}
id.Binding = bind
return bind
}
// spellcheck returns the most likely misspelling of
// the name use.id in the environment use.env.
func (r *resolver) spellcheck(use use) string {
var names []string
// locals
for b := use.env; b != nil; b = b.parent {
for name := range b.bindings {
names = append(names, name)
}
}
// globals
//
// We have no way to enumerate the sets whose membership
// tests are isPredeclared, isUniverse, and isGlobal,
// which includes prior names in the REPL session.
for _, bind := range r.moduleGlobals {
names = append(names, bind.First.Name)
}
sort.Strings(names)
return spell.Nearest(use.id.Name, names)
}
// resolveLocalUses is called when leaving a container (function/module)
// block. It resolves all uses of locals/cells within that block.
func (b *block) resolveLocalUses() {
unresolved := b.uses[:0]
for _, use := range b.uses {
if bind := lookupLocal(use); bind != nil && (bind.Scope == Local || bind.Scope == Cell) {
use.id.Binding = bind
} else {
unresolved = append(unresolved, use)
}
}
b.uses = unresolved
}
func (r *resolver) stmts(stmts []syntax.Stmt) {
for _, stmt := range stmts {
r.stmt(stmt)
}
}
func (r *resolver) stmt(stmt syntax.Stmt) {
switch stmt := stmt.(type) {
case *syntax.ExprStmt:
r.expr(stmt.X)
case *syntax.BranchStmt:
if r.loops == 0 && (stmt.Token == syntax.BREAK || stmt.Token == syntax.CONTINUE) {
r.errorf(stmt.TokenPos, "%s not in a loop", stmt.Token)
}
case *syntax.IfStmt:
if !r.options.TopLevelControl && r.container().function == nil {
r.errorf(stmt.If, "if statement not within a function")
}
r.expr(stmt.Cond)
r.ifstmts++
r.stmts(stmt.True)
r.stmts(stmt.False)
r.ifstmts--
case *syntax.AssignStmt:
r.expr(stmt.RHS)
isAugmented := stmt.Op != syntax.EQ
r.assign(stmt.LHS, isAugmented)
case *syntax.DefStmt:
r.bind(stmt.Name)
fn := &Function{
Name: stmt.Name.Name,
Pos: stmt.Def,
Params: stmt.Params,
Body: stmt.Body,
}
stmt.Function = fn
r.function(fn, stmt.Def)
case *syntax.ForStmt:
if !r.options.TopLevelControl && r.container().function == nil {
r.errorf(stmt.For, "for loop not within a function")
}
r.expr(stmt.X)
const isAugmented = false
r.assign(stmt.Vars, isAugmented)
r.loops++
r.stmts(stmt.Body)
r.loops--
case *syntax.WhileStmt:
if !r.options.While {
r.errorf(stmt.While, doesnt+"support while loops")
}
if !r.options.TopLevelControl && r.container().function == nil {
r.errorf(stmt.While, "while loop not within a function")
}
r.expr(stmt.Cond)
r.loops++
r.stmts(stmt.Body)
r.loops--
case *syntax.ReturnStmt:
if r.container().function == nil {
r.errorf(stmt.Return, "return statement not within a function")
}
if stmt.Result != nil {
r.expr(stmt.Result)
}
case *syntax.LoadStmt:
// A load statement may not be nested in any other statement.
if r.container().function != nil {
r.errorf(stmt.Load, "load statement within a function")
} else if r.loops > 0 {
r.errorf(stmt.Load, "load statement within a loop")
} else if r.ifstmts > 0 {
r.errorf(stmt.Load, "load statement within a conditional")
}
for i, from := range stmt.From {
if from.Name == "" {
r.errorf(from.NamePos, "load: empty identifier")
continue
}
if from.Name[0] == '_' {
r.errorf(from.NamePos, "load: names with leading underscores are not exported: %s", from.Name)
}
id := stmt.To[i]
if r.options.LoadBindsGlobally {
r.bind(id)
} else if r.bindLocal(id) && !r.options.GlobalReassign {
// "Global" in AllowGlobalReassign is a misnomer for "toplevel".
// Sadly we can't report the previous declaration
// as id.Binding may not be set yet.
r.errorf(id.NamePos, "cannot reassign top-level %s", id.Name)
}
}
default:
log.Panicf("unexpected stmt %T", stmt)
}
}
func (r *resolver) assign(lhs syntax.Expr, isAugmented bool) {
switch lhs := lhs.(type) {
case *syntax.Ident:
// x = ...
r.bind(lhs)
case *syntax.IndexExpr:
// x[i] = ...
r.expr(lhs.X)
r.expr(lhs.Y)
case *syntax.DotExpr:
// x.f = ...
r.expr(lhs.X)
case *syntax.TupleExpr:
// (x, y) = ...
if isAugmented {
r.errorf(syntax.Start(lhs), "can't use tuple expression in augmented assignment")
}
for _, elem := range lhs.List {
r.assign(elem, isAugmented)
}
case *syntax.ListExpr:
// [x, y, z] = ...
if isAugmented {
r.errorf(syntax.Start(lhs), "can't use list expression in augmented assignment")
}
for _, elem := range lhs.List {
r.assign(elem, isAugmented)
}
case *syntax.ParenExpr:
r.assign(lhs.X, isAugmented)
default:
name := strings.ToLower(strings.TrimPrefix(fmt.Sprintf("%T", lhs), "*syntax."))
r.errorf(syntax.Start(lhs), "can't assign to %s", name)
}
}
func (r *resolver) expr(e syntax.Expr) {
switch e := e.(type) {
case *syntax.Ident:
r.use(e)
case *syntax.Literal:
case *syntax.ListExpr:
for _, x := range e.List {
r.expr(x)
}
case *syntax.CondExpr:
r.expr(e.Cond)
r.expr(e.True)
r.expr(e.False)
case *syntax.IndexExpr:
r.expr(e.X)
r.expr(e.Y)
case *syntax.DictEntry:
r.expr(e.Key)
r.expr(e.Value)
case *syntax.SliceExpr:
r.expr(e.X)
if e.Lo != nil {
r.expr(e.Lo)
}
if e.Hi != nil {
r.expr(e.Hi)
}
if e.Step != nil {
r.expr(e.Step)
}
case *syntax.Comprehension:
// The 'in' operand of the first clause (always a ForClause)
// is resolved in the outer block; consider: [x for x in x].
clause := e.Clauses[0].(*syntax.ForClause)
r.expr(clause.X)
// A list/dict comprehension defines a new lexical block.
// Locals defined within the block will be allotted
// distinct slots in the locals array of the innermost
// enclosing container (function/module) block.
r.push(&block{comp: e})
const isAugmented = false
r.assign(clause.Vars, isAugmented)
for _, clause := range e.Clauses[1:] {
switch clause := clause.(type) {
case *syntax.IfClause:
r.expr(clause.Cond)
case *syntax.ForClause:
r.assign(clause.Vars, isAugmented)
r.expr(clause.X)
}
}
r.expr(e.Body) // body may be *DictEntry
r.pop()
case *syntax.TupleExpr:
for _, x := range e.List {
r.expr(x)
}
case *syntax.DictExpr:
for _, entry := range e.List {
entry := entry.(*syntax.DictEntry)
r.expr(entry.Key)
r.expr(entry.Value)
}
case *syntax.UnaryExpr:
r.expr(e.X)
case *syntax.BinaryExpr:
r.expr(e.X)
r.expr(e.Y)
case *syntax.DotExpr:
r.expr(e.X)
// ignore e.Name
case *syntax.CallExpr:
r.expr(e.Fn)
var seenVarargs, seenKwargs bool
var seenName map[string]bool
var n, p int
for _, arg := range e.Args {
pos, _ := arg.Span()
if unop, ok := arg.(*syntax.UnaryExpr); ok && unop.Op == syntax.STARSTAR {
// **kwargs
if seenKwargs {
r.errorf(pos, "multiple **kwargs not allowed")
}
seenKwargs = true
r.expr(arg)
} else if ok && unop.Op == syntax.STAR {
// *args
if seenKwargs {
r.errorf(pos, "*args may not follow **kwargs")
} else if seenVarargs {
r.errorf(pos, "multiple *args not allowed")
}
seenVarargs = true
r.expr(arg)
} else if binop, ok := arg.(*syntax.BinaryExpr); ok && binop.Op == syntax.EQ {
// k=v
n++
if seenKwargs {
r.errorf(pos, "keyword argument may not follow **kwargs")
} else if seenVarargs {
r.errorf(pos, "keyword argument may not follow *args")
}
x := binop.X.(*syntax.Ident)
if seenName[x.Name] {
r.errorf(x.NamePos, "keyword argument %q is repeated", x.Name)
} else {
if seenName == nil {
seenName = make(map[string]bool)
}
seenName[x.Name] = true
}
r.expr(binop.Y)
} else {
// positional argument
p++
if seenVarargs {
r.errorf(pos, "positional argument may not follow *args")
} else if seenKwargs {
r.errorf(pos, "positional argument may not follow **kwargs")
} else if len(seenName) > 0 {
r.errorf(pos, "positional argument may not follow named")
}
r.expr(arg)
}
}
// Fail gracefully if compiler-imposed limit is exceeded.
if p >= 256 {
pos, _ := e.Span()
r.errorf(pos, "%v positional arguments in call, limit is 255", p)
}
if n >= 256 {
pos, _ := e.Span()
r.errorf(pos, "%v keyword arguments in call, limit is 255", n)
}
case *syntax.LambdaExpr:
fn := &Function{
Name: "lambda",
Pos: e.Lambda,
Params: e.Params,
Body: []syntax.Stmt{&syntax.ReturnStmt{Result: e.Body}},
}
e.Function = fn
r.function(fn, e.Lambda)
case *syntax.ParenExpr:
r.expr(e.X)
default:
log.Panicf("unexpected expr %T", e)
}
}
func (r *resolver) function(function *Function, pos syntax.Position) {
// Resolve defaults in enclosing environment.
for _, param := range function.Params {
if binary, ok := param.(*syntax.BinaryExpr); ok {
r.expr(binary.Y)
}
}
// Enter function block.
b := &block{function: function}
r.push(b)
var seenOptional bool
var star *syntax.UnaryExpr // * or *args param
var starStar *syntax.Ident // **kwargs ident
var numKwonlyParams int
for _, param := range function.Params {
switch param := param.(type) {
case *syntax.Ident:
// e.g. x
if starStar != nil {
r.errorf(param.NamePos, "required parameter may not follow **%s", starStar.Name)
} else if star != nil {
numKwonlyParams++
} else if seenOptional {
r.errorf(param.NamePos, "required parameter may not follow optional")
}
if r.bind(param) {
r.errorf(param.NamePos, "duplicate parameter: %s", param.Name)
}
case *syntax.BinaryExpr:
// e.g. y=dflt
if starStar != nil {
r.errorf(param.OpPos, "optional parameter may not follow **%s", starStar.Name)
} else if star != nil {
numKwonlyParams++
}
if id := param.X.(*syntax.Ident); r.bind(id) {
r.errorf(param.OpPos, "duplicate parameter: %s", id.Name)
}
seenOptional = true
case *syntax.UnaryExpr:
// * or *args or **kwargs
if param.Op == syntax.STAR {
if starStar != nil {
r.errorf(param.OpPos, "* parameter may not follow **%s", starStar.Name)
} else if star != nil {
r.errorf(param.OpPos, "multiple * parameters not allowed")
} else {
star = param
}
} else {
if starStar != nil {
r.errorf(param.OpPos, "multiple ** parameters not allowed")
}
starStar = param.X.(*syntax.Ident)
}
}
}
// Bind the *args and **kwargs parameters at the end,
// so that regular parameters a/b/c are contiguous and
// there is no hole for the "*":
// def f(a, b, *args, c=0, **kwargs)
// def f(a, b, *, c=0, **kwargs)
if star != nil {
if id, _ := star.X.(*syntax.Ident); id != nil {
// *args
if r.bind(id) {
r.errorf(id.NamePos, "duplicate parameter: %s", id.Name)
}
function.HasVarargs = true
} else if numKwonlyParams == 0 {
r.errorf(star.OpPos, "bare * must be followed by keyword-only parameters")
}
}
if starStar != nil {
if r.bind(starStar) {
r.errorf(starStar.NamePos, "duplicate parameter: %s", starStar.Name)
}
function.HasKwargs = true
}
function.NumKwonlyParams = numKwonlyParams
r.stmts(function.Body)
// Resolve all uses of this function's local vars,
// and keep just the remaining uses of free/global vars.
b.resolveLocalUses()
// Leave function block.
r.pop()
// References within the function body to globals are not
// resolved until the end of the module.
}
func (r *resolver) resolveNonLocalUses(b *block) {
// First resolve inner blocks.
for _, child := range b.children {
r.resolveNonLocalUses(child)
}
for _, use := range b.uses {
use.id.Binding = r.lookupLexical(use, use.env)
}
}
// lookupLocal looks up an identifier within its immediately enclosing function.
func lookupLocal(use use) *Binding {
for env := use.env; env != nil; env = env.parent {
if bind, ok := env.bindings[use.id.Name]; ok {
if bind.Scope == Free {
// shouldn't exist till later
log.Panicf("%s: internal error: %s, %v", use.id.NamePos, use.id.Name, bind)
}
return bind // found
}
if env.function != nil {
break
}
}
return nil // not found in this function
}
// lookupLexical looks up an identifier use.id within its lexically enclosing environment.
// The use.env field captures the original environment for error reporting.
func (r *resolver) lookupLexical(use use, env *block) (bind *Binding) {
if debug {
fmt.Printf("lookupLexical %s in %s = ...\n", use.id.Name, env)
defer func() { fmt.Printf("= %v\n", bind) }()
}
// Is this the file block?
if env == r.file {
return r.useToplevel(use) // file-local, global, predeclared, or not found
}
// Defined in this block?
bind, ok := env.bindings[use.id.Name]
if !ok {
// Defined in parent block?
bind = r.lookupLexical(use, env.parent)
if env.function != nil && (bind.Scope == Local || bind.Scope == Free || bind.Scope == Cell) {
// Found in parent block, which belongs to enclosing function.
// Add the parent's binding to the function's freevars,
// and add a new 'free' binding to the inner function's block,
// and turn the parent's local into cell.
if bind.Scope == Local {
bind.Scope = Cell
}
index := len(env.function.FreeVars)
env.function.FreeVars = append(env.function.FreeVars, bind)
bind = &Binding{
First: bind.First,
Scope: Free,
Index: index,
}
if debug {
fmt.Printf("creating freevar %v in function at %s: %s\n",
len(env.function.FreeVars), env.function.Pos, use.id.Name)
}
}
// Memoize, to avoid duplicate free vars
// and redundant global (failing) lookups.
env.bind(use.id.Name, bind)
}
return bind
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/resolve/resolve_test.go
+92
View File
@@ -0,0 +1,92 @@
// Copyright 2017 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package resolve_test
import (
"strings"
"testing"
"go.starlark.net/internal/chunkedfile"
"go.starlark.net/resolve"
"go.starlark.net/starlarktest"
"go.starlark.net/syntax"
)
// A test may enable non-standard options by containing (e.g.) "option:recursion".
func getOptions(src string) *syntax.FileOptions {
return &syntax.FileOptions{
Set: option(src, "set"),
While: option(src, "while"),
TopLevelControl: option(src, "toplevelcontrol"),
GlobalReassign: option(src, "globalreassign"),
LoadBindsGlobally: option(src, "loadbindsglobally"),
Recursion: option(src, "recursion"),
}
}
func option(chunk, name string) bool {
return strings.Contains(chunk, "option:"+name)
}
func TestResolve(t *testing.T) {
filename := starlarktest.DataFile("resolve", "testdata/resolve.star")
for _, chunk := range chunkedfile.Read(filename, t) {
// A chunk may set options by containing e.g. "option:recursion".
opts := getOptions(chunk.Source)
f, err := opts.Parse(filename, chunk.Source, 0)
if err != nil {
t.Error(err)
continue
}
if err := resolve.File(f, isPredeclared, isUniversal); err != nil {
for _, err := range err.(resolve.ErrorList) {
chunk.GotError(int(err.Pos.Line), err.Msg)
}
}
chunk.Done()
}
}
func TestDefVarargsAndKwargsSet(t *testing.T) {
source := "def f(*args, **kwargs): pass\n"
file, err := syntax.Parse("foo.star", source, 0)
if err != nil {
t.Fatal(err)
}
if err := resolve.File(file, isPredeclared, isUniversal); err != nil {
t.Fatal(err)
}
fn := file.Stmts[0].(*syntax.DefStmt).Function.(*resolve.Function)
if !fn.HasVarargs {
t.Error("HasVarargs not set")
}
if !fn.HasKwargs {
t.Error("HasKwargs not set")
}
}
func TestLambdaVarargsAndKwargsSet(t *testing.T) {
source := "f = lambda *args, **kwargs: 0\n"
file, err := syntax.Parse("foo.star", source, 0)
if err != nil {
t.Fatal(err)
}
if err := resolve.File(file, isPredeclared, isUniversal); err != nil {
t.Fatal(err)
}
lam := file.Stmts[0].(*syntax.AssignStmt).RHS.(*syntax.LambdaExpr).Function.(*resolve.Function)
if !lam.HasVarargs {
t.Error("HasVarargs not set")
}
if !lam.HasKwargs {
t.Error("HasKwargs not set")
}
}
func isPredeclared(name string) bool { return name == "M" }
func isUniversal(name string) bool { return name == "U" || name == "float" }
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/resolve/testdata/resolve.star
Vendored
+383
View File
@@ -0,0 +1,383 @@
# Tests of resolver errors.
#
# The initial environment contains the predeclared names "M"
# (module-specific) and "U" (universal). This distinction
# should be unobservable to the Starlark program.
# use of declared global
x = 1
_ = x
---
# premature use of global is not a static error;
# see github.com/google/skylark/issues/116.
_ = x
x = 1
---
# use of undefined global
_ = x ### "undefined: x"
---
# redeclaration of global
x = 1
x = 2 ### "cannot reassign global x declared at .*resolve.star:23:1"
---
# Redeclaration of predeclared names is allowed.
#
# This rule permits tool maintainers to add members to the predeclared
# environment without breaking existing programs.
# module-specific predeclared name
M = 1 # ok
M = 2 ### "cannot reassign global M declared at .*/resolve.star"
# universal predeclared name
U = 1 # ok
U = 1 ### "cannot reassign global U declared at .*/resolve.star"
---
# A global declaration shadows all references to a predeclared;
# see github.com/google/skylark/issues/116.
a = U # ok: U is a reference to the global defined on the next line.
U = 1
---
# reference to predeclared name
M()
---
# locals may be referenced before they are defined
def f():
M(x) # dynamic error
x = 1
---
# Various forms of assignment:
def f(x): # parameter
M(x)
M(y) ### "undefined: y"
(a, b) = 1, 2
M(a)
M(b)
M(c) ### "undefined: c"
[p, q] = 1, 2
M(p)
M(q)
M(r) ### "undefined: r"
---
# a comprehension introduces a separate lexical block
_ = [x for x in "abc"]
M(x) ### "undefined: x"
---
# Functions may have forward refs.
def f():
g()
h() ### "undefined: h"
def inner():
i()
i = lambda: 0
def g():
f()
---
# It is not permitted to rebind a global using a += assignment.
x = [1]
x.extend([2]) # ok
x += [3] ### `cannot reassign global x`
def f():
x += [4] # x is local to f
y = 1
y += 2 ### `cannot reassign global y`
z += 3 # ok (but fails dynamically because z is undefined)
---
def f(a):
if 1==1:
b = 1
c = 1
M(a) # ok: param
M(b) # ok: maybe bound local
M(c) # ok: bound local
M(d) # NB: we don't do a use-before-def check on local vars!
M(e) # ok: global
M(f) # ok: global
d = 1
e = 1
---
# This program should resolve successfully but fail dynamically.
x = 1
def f():
M(x) # dynamic error: reference to undefined local
x = 2
f()
---
load("module", "name") # ok
def f():
load("foo", "bar") ### "load statement within a function"
load("foo",
"", ### "load: empty identifier"
"_a", ### "load: names with leading underscores are not exported: _a"
b="", ### "load: empty identifier"
c="_d", ### "load: names with leading underscores are not exported: _d"
_e="f") # ok
---
# option:toplevelcontrol
if M:
load("foo", "bar") ### "load statement within a conditional"
---
# option:toplevelcontrol
for x in M:
load("foo", "bar") ### "load statement within a loop"
---
# option:toplevelcontrol option:while
while M:
load("foo", "bar") ### "load statement within a loop"
---
# return statements must be within a function
return ### "return statement not within a function"
---
# if-statements and for-loops at top-level are forbidden
# (without globalreassign option)
for x in "abc": ### "for loop not within a function"
pass
if x: ### "if statement not within a function"
pass
---
# option:toplevelcontrol
for x in "abc": # ok
pass
if x: # ok
pass
---
# while loops are forbidden (without -recursion option)
def f():
while U: ### "dialect does not support while loops"
pass
---
# option:while
def f():
while U: # ok
pass
while U: ### "while loop not within a function"
pass
---
# option:toplevelcontrol option:while
while U: # ok
pass
---
# The parser allows any expression on the LHS of an assignment.
1 = 0 ### "can't assign to literal"
1+2 = 0 ### "can't assign to binaryexpr"
f() = 0 ### "can't assign to callexpr"
[a, b] = 0
[c, d] += 0 ### "can't use list expression in augmented assignment"
(e, f) += 0 ### "can't use tuple expression in augmented assignment"
[] = 0 # ok
() = 0 # ok
---
# break and continue statements must appear within a loop
break ### "break not in a loop"
continue ### "continue not in a loop"
pass
---
# Positional arguments (and required parameters)
# must appear before named arguments (and optional parameters).
M(x=1, 2) ### `positional argument may not follow named`
def f(x=1, y): pass ### `required parameter may not follow optional`
---
# No parameters may follow **kwargs in a declaration.
def f(**kwargs, x): ### `parameter may not follow \*\*kwargs`
pass
def g(**kwargs, *args): ### `\* parameter may not follow \*\*kwargs`
pass
def h(**kwargs1, **kwargs2): ### `multiple \*\* parameters not allowed`
pass
---
# Only keyword-only params and **kwargs may follow *args in a declaration.
def f(*args, x): # ok
pass
def g(*args1, *args2): ### `multiple \* parameters not allowed`
pass
def h(*, ### `bare \* must be followed by keyword-only parameters`
*): ### `multiple \* parameters not allowed`
pass
def i(*args, *): ### `multiple \* parameters not allowed`
pass
def j(*, ### `bare \* must be followed by keyword-only parameters`
*args): ### `multiple \* parameters not allowed`
pass
def k(*, **kwargs): ### `bare \* must be followed by keyword-only parameters`
pass
def l(*): ### `bare \* must be followed by keyword-only parameters`
pass
def m(*args, a=1, **kwargs): # ok
pass
def n(*, a=1, **kwargs): # ok
pass
---
# No arguments may follow **kwargs in a call.
def f(*args, **kwargs):
pass
f(**{}, 1) ### `argument may not follow \*\*kwargs`
f(**{}, x=1) ### `argument may not follow \*\*kwargs`
f(**{}, *[]) ### `\*args may not follow \*\*kwargs`
f(**{}, **{}) ### `multiple \*\*kwargs not allowed`
---
# Only **kwargs may follow *args in a call.
def f(*args, **kwargs):
pass
f(*[], 1) ### `positional argument may not follow \*args`
f(*[], a=1) ### `keyword argument may not follow \*args`
f(*[], *[]) ### `multiple \*args not allowed`
f(*[], **{}) # ok
---
# Parameter names must be unique.
def f(a, b, a): pass ### "duplicate parameter: a"
def g(args, b, *args): pass ### "duplicate parameter: args"
def h(kwargs, a, **kwargs): pass ### "duplicate parameter: kwargs"
def i(*x, **x): pass ### "duplicate parameter: x"
---
# Floating-point support is now standard.
a = float("3.141")
b = 1 / 2
c = 3.141
---
# option:globalreassign
# Legacy Bazel (and Python) semantics: def must precede use even for globals.
_ = x ### `undefined: x`
x = 1
---
# option:globalreassign
# Legacy Bazel (and Python) semantics: reassignment of globals is allowed.
x = 1
x = 2 # ok
---
# option:globalreassign
# Redeclaration of predeclared names is allowed.
# module-specific predeclared name
M = 1 # ok
M = 2 # ok (legacy)
# universal predeclared name
U = 1 # ok
U = 1 # ok (legacy)
---
# https://github.com/bazelbuild/starlark/starlark/issues/21
def f(**kwargs): pass
f(a=1, a=1) ### `keyword argument "a" is repeated`
---
# spelling
print = U
hello = 1
print(hollo) ### `undefined: hollo \(did you mean hello\?\)`
def f(abc):
print(abd) ### `undefined: abd \(did you mean abc\?\)`
print(goodbye) ### `undefined: goodbye$`
---
load("module", "x") # ok
x = 1 ### `cannot reassign local x`
load("module", "x") ### `cannot reassign top-level x`
---
# option:loadbindsglobally
load("module", "x") # ok
x = 1 ### `cannot reassign global x`
load("module", "x") ### `cannot reassign global x`
---
# option:globalreassign
load("module", "x") # ok
x = 1 # ok
load("module", "x") # ok
---
# option:globalreassign option:loadbindsglobally
load("module", "x") # ok
x = 1
load("module", "x") # ok
---
_ = x # forward ref to file-local
load("module", "x") # ok
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/bench_test.go
+170
View File
@@ -0,0 +1,170 @@
// Copyright 2018 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package starlark_test
import (
"bytes"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"go.starlark.net/lib/json"
"go.starlark.net/starlark"
"go.starlark.net/starlarktest"
)
func BenchmarkStarlark(b *testing.B) {
starlark.Universe["json"] = json.Module
testdata := starlarktest.DataFile("starlark", ".")
thread := new(starlark.Thread)
for _, file := range []string{
"testdata/benchmark.star",
// ...
} {
filename := filepath.Join(testdata, file)
src, err := os.ReadFile(filename)
if err != nil {
b.Error(err)
continue
}
opts := getOptions(string(src))
// Evaluate the file once.
globals, err := starlark.ExecFileOptions(opts, thread, filename, src, nil)
if err != nil {
reportEvalError(b, err)
}
// Repeatedly call each global function named bench_* as a benchmark.
for _, name := range globals.Keys() {
value := globals[name]
if fn, ok := value.(*starlark.Function); ok && strings.HasPrefix(name, "bench_") {
b.Run(name, func(b *testing.B) {
_, err := starlark.Call(thread, fn, starlark.Tuple{benchmark{b}}, nil)
if err != nil {
reportEvalError(b, err)
}
})
}
}
}
}
// A benchmark is passed to each bench_xyz(b) function in a bench_*.star file.
// It provides b.n, the number of iterations that must be executed by the function,
// which is typically of the form:
//
// def bench_foo(b):
// for _ in range(b.n):
// ...work...
//
// It also provides stop, start, and restart methods to stop the clock in case
// there is significant set-up work that should not count against the measured
// operation.
//
// (This interface is inspired by Go's testing.B, and is also implemented
// by the java.starlark.net implementation; see
// https://github.com/bazelbuild/starlark/pull/75#pullrequestreview-275604129.)
type benchmark struct {
b *testing.B
}
func (benchmark) Freeze() {}
func (benchmark) Truth() starlark.Bool { return true }
func (benchmark) Type() string { return "benchmark" }
func (benchmark) String() string { return "<benchmark>" }
func (benchmark) Hash() (uint32, error) { return 0, fmt.Errorf("unhashable: benchmark") }
func (benchmark) AttrNames() []string { return []string{"n", "restart", "start", "stop"} }
func (b benchmark) Attr(name string) (starlark.Value, error) {
switch name {
case "n":
return starlark.MakeInt(b.b.N), nil
case "restart":
return benchmarkRestart.BindReceiver(b), nil
case "start":
return benchmarkStart.BindReceiver(b), nil
case "stop":
return benchmarkStop.BindReceiver(b), nil
}
return nil, nil
}
var (
benchmarkRestart = starlark.NewBuiltin("restart", benchmarkRestartImpl)
benchmarkStart = starlark.NewBuiltin("start", benchmarkStartImpl)
benchmarkStop = starlark.NewBuiltin("stop", benchmarkStopImpl)
)
func benchmarkRestartImpl(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
b.Receiver().(benchmark).b.ResetTimer()
return starlark.None, nil
}
func benchmarkStartImpl(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
b.Receiver().(benchmark).b.StartTimer()
return starlark.None, nil
}
func benchmarkStopImpl(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
b.Receiver().(benchmark).b.StopTimer()
return starlark.None, nil
}
// BenchmarkProgram measures operations relevant to compiled programs.
// TODO(adonovan): use a bigger testdata program.
func BenchmarkProgram(b *testing.B) {
// Measure time to read a source file (approx 600us but depends on hardware and file system).
filename := starlarktest.DataFile("starlark", "testdata/paths.star")
var src []byte
b.Run("read", func(b *testing.B) {
for i := 0; i < b.N; i++ {
var err error
src, err = os.ReadFile(filename)
if err != nil {
b.Fatal(err)
}
}
})
// Measure time to turn a source filename into a compiled program (approx 450us).
var prog *starlark.Program
b.Run("compile", func(b *testing.B) {
for i := 0; i < b.N; i++ {
var err error
_, prog, err = starlark.SourceProgram(filename, src, starlark.StringDict(nil).Has)
if err != nil {
b.Fatal(err)
}
}
})
// Measure time to encode a compiled program to a memory buffer
// (approx 20us; was 75-120us with gob encoding).
var out bytes.Buffer
b.Run("encode", func(b *testing.B) {
for i := 0; i < b.N; i++ {
out.Reset()
if err := prog.Write(&out); err != nil {
b.Fatal(err)
}
}
})
// Measure time to decode a compiled program from a memory buffer
// (approx 20us; was 135-250us with gob encoding)
b.Run("decode", func(b *testing.B) {
for i := 0; i < b.N; i++ {
in := bytes.NewReader(out.Bytes())
if _, err := starlark.CompiledProgram(in); err != nil {
b.Fatal(err)
}
}
})
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/debug.go
+42
View File
@@ -0,0 +1,42 @@
package starlark
import "go.starlark.net/syntax"
// This file defines an experimental API for the debugging tools.
// Some of these declarations expose details of internal packages.
// (The debugger makes liberal use of exported fields of unexported types.)
// Breaking changes may occur without notice.
// Local returns the value of the i'th local variable.
// It may be nil if not yet assigned.
//
// Local may be called only for frames whose Callable is a *Function (a
// function defined by Starlark source code), and only while the frame
// is active; it will panic otherwise.
//
// This function is provided only for debugging tools.
//
// THIS API IS EXPERIMENTAL AND MAY CHANGE WITHOUT NOTICE.
func (fr *frame) Local(i int) Value { return fr.locals[i] }
// DebugFrame is the debugger API for a frame of the interpreter's call stack.
//
// Most applications have no need for this API; use CallFrame instead.
//
// Clients must not retain a DebugFrame nor call any of its methods once
// the current built-in call has returned or execution has resumed
// after a breakpoint as this may have unpredictable effects, including
// but not limited to retention of object that would otherwise be garbage.
type DebugFrame interface {
Callable() Callable // returns the frame's function
Local(i int) Value // returns the value of the (Starlark) frame's ith local variable
Position() syntax.Position // returns the current position of execution in this frame
}
// DebugFrame returns the debugger interface for
// the specified frame of the interpreter's call stack.
// Frame numbering is as for Thread.CallFrame.
//
// This function is intended for use in debugging tools.
// Most applications should have no need for it; use CallFrame instead.
func (thread *Thread) DebugFrame(depth int) DebugFrame { return thread.frameAt(depth) }
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/empty.s
+3
View File
@@ -0,0 +1,3 @@
// The presence of this file allows the package to use the
// "go:linkname" hack to call non-exported functions in the
// Go runtime, such as hardware-accelerated string hashing.
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/eval.go
+1667
View File
File diff suppressed because it is too large Load Diff
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/eval_test.go
+1058
View File
File diff suppressed because it is too large Load Diff
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/example_test.go
+322
View File
@@ -0,0 +1,322 @@
// Copyright 2017 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package starlark_test
import (
"fmt"
"log"
"reflect"
"sort"
"strings"
"sync"
"sync/atomic"
"testing"
"unsafe"
"go.starlark.net/starlark"
)
// ExampleExecFile demonstrates a simple embedding
// of the Starlark interpreter into a Go program.
func ExampleExecFile() {
const data = `
print(greeting + ", world")
print(repeat("one"))
print(repeat("mur", 2))
squares = [x*x for x in range(10)]
`
// repeat(str, n=1) is a Go function called from Starlark.
// It behaves like the 'string * int' operation.
repeat := func(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var s string
var n int = 1
if err := starlark.UnpackArgs(b.Name(), args, kwargs, "s", &s, "n?", &n); err != nil {
return nil, err
}
return starlark.String(strings.Repeat(s, n)), nil
}
// The Thread defines the behavior of the built-in 'print' function.
thread := &starlark.Thread{
Name: "example",
Print: func(_ *starlark.Thread, msg string) { fmt.Println(msg) },
}
// This dictionary defines the pre-declared environment.
predeclared := starlark.StringDict{
"greeting": starlark.String("hello"),
"repeat": starlark.NewBuiltin("repeat", repeat),
}
// Execute a program.
globals, err := starlark.ExecFile(thread, "apparent/filename.star", data, predeclared)
if err != nil {
if evalErr, ok := err.(*starlark.EvalError); ok {
log.Fatal(evalErr.Backtrace())
}
log.Fatal(err)
}
// Print the global environment.
fmt.Println("\nGlobals:")
for _, name := range globals.Keys() {
v := globals[name]
fmt.Printf("%s (%s) = %s\n", name, v.Type(), v.String())
}
// Output:
// hello, world
// one
// murmur
//
// Globals:
// squares (list) = [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
}
// ExampleThread_Load_sequential demonstrates a simple caching
// implementation of 'load' that works sequentially.
func ExampleThread_Load_sequential() {
fakeFilesystem := map[string]string{
"c.star": `load("b.star", "b"); c = b + "!"`,
"b.star": `load("a.star", "a"); b = a + ", world"`,
"a.star": `a = "Hello"`,
}
type entry struct {
globals starlark.StringDict
err error
}
cache := make(map[string]*entry)
var load func(_ *starlark.Thread, module string) (starlark.StringDict, error)
load = func(_ *starlark.Thread, module string) (starlark.StringDict, error) {
e, ok := cache[module]
if e == nil {
if ok {
// request for package whose loading is in progress
return nil, fmt.Errorf("cycle in load graph")
}
// Add a placeholder to indicate "load in progress".
cache[module] = nil
// Load and initialize the module in a new thread.
data := fakeFilesystem[module]
thread := &starlark.Thread{Name: "exec " + module, Load: load}
globals, err := starlark.ExecFile(thread, module, data, nil)
e = &entry{globals, err}
// Update the cache.
cache[module] = e
}
return e.globals, e.err
}
globals, err := load(nil, "c.star")
if err != nil {
log.Fatal(err)
}
fmt.Println(globals["c"])
// Output:
// "Hello, world!"
}
// ExampleThread_Load_parallel demonstrates a parallel implementation
// of 'load' with caching, duplicate suppression, and cycle detection.
func ExampleThread_Load_parallel() {
cache := &cache{
cache: make(map[string]*entry),
fakeFilesystem: map[string]string{
"c.star": `load("a.star", "a"); c = a * 2`,
"b.star": `load("a.star", "a"); b = a * 3`,
"a.star": `a = 1; print("loaded a")`,
},
}
// We load modules b and c in parallel by concurrent calls to
// cache.Load. Both of them load module a, but a is executed
// only once, as witnessed by the sole output of its print
// statement.
ch := make(chan string)
for _, name := range []string{"b", "c"} {
go func(name string) {
globals, err := cache.Load(name + ".star")
if err != nil {
log.Fatal(err)
}
ch <- fmt.Sprintf("%s = %s", name, globals[name])
}(name)
}
got := []string{<-ch, <-ch}
sort.Strings(got)
fmt.Println(strings.Join(got, "\n"))
// Output:
// loaded a
// b = 3
// c = 2
}
// TestThread_Load_parallelCycle demonstrates detection
// of cycles during parallel loading.
func TestThreadLoad_ParallelCycle(t *testing.T) {
cache := &cache{
cache: make(map[string]*entry),
fakeFilesystem: map[string]string{
"c.star": `load("b.star", "b"); c = b * 2`,
"b.star": `load("a.star", "a"); b = a * 3`,
"a.star": `load("c.star", "c"); a = c * 5; print("loaded a")`,
},
}
ch := make(chan string)
for _, name := range "bc" {
name := string(name)
go func() {
_, err := cache.Load(name + ".star")
if err == nil {
log.Fatalf("Load of %s.star succeeded unexpectedly", name)
}
ch <- err.Error()
}()
}
got := []string{<-ch, <-ch}
sort.Strings(got)
// Typically, the c goroutine quickly blocks behind b;
// b loads a, and a then fails to load c because it forms a cycle.
// The errors observed by the two goroutines are:
want1 := []string{
"cannot load a.star: cannot load c.star: cycle in load graph", // from b
"cannot load b.star: cannot load a.star: cannot load c.star: cycle in load graph", // from c
}
// But if the c goroutine is slow to start, b loads a,
// and a loads c; then c fails to load b because it forms a cycle.
// The errors this time are:
want2 := []string{
"cannot load a.star: cannot load c.star: cannot load b.star: cycle in load graph", // from b
"cannot load b.star: cycle in load graph", // from c
}
if !reflect.DeepEqual(got, want1) && !reflect.DeepEqual(got, want2) {
t.Error(got)
}
}
// cache is a concurrency-safe, duplicate-suppressing,
// non-blocking cache of the doLoad function.
// See Section 9.7 of gopl.io for an explanation of this structure.
// It also features online deadlock (load cycle) detection.
type cache struct {
cacheMu sync.Mutex
cache map[string]*entry
fakeFilesystem map[string]string
}
type entry struct {
owner unsafe.Pointer // a *cycleChecker; see cycleCheck
globals starlark.StringDict
err error
ready chan struct{}
}
func (c *cache) Load(module string) (starlark.StringDict, error) {
return c.get(new(cycleChecker), module)
}
// get loads and returns an entry (if not already loaded).
func (c *cache) get(cc *cycleChecker, module string) (starlark.StringDict, error) {
c.cacheMu.Lock()
e := c.cache[module]
if e != nil {
c.cacheMu.Unlock()
// Some other goroutine is getting this module.
// Wait for it to become ready.
// Detect load cycles to avoid deadlocks.
if err := cycleCheck(e, cc); err != nil {
return nil, err
}
cc.setWaitsFor(e)
<-e.ready
cc.setWaitsFor(nil)
} else {
// First request for this module.
e = &entry{ready: make(chan struct{})}
c.cache[module] = e
c.cacheMu.Unlock()
e.setOwner(cc)
e.globals, e.err = c.doLoad(cc, module)
e.setOwner(nil)
// Broadcast that the entry is now ready.
close(e.ready)
}
return e.globals, e.err
}
func (c *cache) doLoad(cc *cycleChecker, module string) (starlark.StringDict, error) {
thread := &starlark.Thread{
Name: "exec " + module,
Print: func(_ *starlark.Thread, msg string) { fmt.Println(msg) },
Load: func(_ *starlark.Thread, module string) (starlark.StringDict, error) {
// Tunnel the cycle-checker state for this "thread of loading".
return c.get(cc, module)
},
}
data := c.fakeFilesystem[module]
return starlark.ExecFile(thread, module, data, nil)
}
// -- concurrent cycle checking --
// A cycleChecker is used for concurrent deadlock detection.
// Each top-level call to Load creates its own cycleChecker,
// which is passed to all recursive calls it makes.
// It corresponds to a logical thread in the deadlock detection literature.
type cycleChecker struct {
waitsFor unsafe.Pointer // an *entry; see cycleCheck
}
func (cc *cycleChecker) setWaitsFor(e *entry) {
atomic.StorePointer(&cc.waitsFor, unsafe.Pointer(e))
}
func (e *entry) setOwner(cc *cycleChecker) {
atomic.StorePointer(&e.owner, unsafe.Pointer(cc))
}
// cycleCheck reports whether there is a path in the waits-for graph
// from resource 'e' to thread 'me'.
//
// The waits-for graph (WFG) is a bipartite graph whose nodes are
// alternately of type entry and cycleChecker. Each node has at most
// one outgoing edge. An entry has an "owner" edge to a cycleChecker
// while it is being readied by that cycleChecker, and a cycleChecker
// has a "waits-for" edge to an entry while it is waiting for that entry
// to become ready.
//
// Before adding a waits-for edge, the cache checks whether the new edge
// would form a cycle. If so, this indicates that the load graph is
// cyclic and that the following wait operation would deadlock.
func cycleCheck(e *entry, me *cycleChecker) error {
for e != nil {
cc := (*cycleChecker)(atomic.LoadPointer(&e.owner))
if cc == nil {
break
}
if cc == me {
return fmt.Errorf("cycle in load graph")
}
e = (*entry)(atomic.LoadPointer(&cc.waitsFor))
}
return nil
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/hashtable.go
+442
View File
@@ -0,0 +1,442 @@
// Copyright 2017 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package starlark
import (
"fmt"
"math/big"
_ "unsafe" // for go:linkname hack
)
// hashtable is used to represent Starlark dict and set values.
// It is a hash table whose key/value entries form a doubly-linked list
// in the order the entries were inserted.
//
// Initialized instances of hashtable must not be copied.
type hashtable struct {
table []bucket // len is zero or a power of two
bucket0 [1]bucket // inline allocation for small maps.
len uint32
itercount uint32 // number of active iterators (ignored if frozen)
head *entry // insertion order doubly-linked list; may be nil
tailLink **entry // address of nil link at end of list (perhaps &head)
frozen bool
_ noCopy // triggers vet copylock check on this type.
}
// noCopy is zero-sized type that triggers vet's copylock check.
// See https://github.com/golang/go/issues/8005#issuecomment-190753527.
type noCopy struct{}
func (*noCopy) Lock() {}
func (*noCopy) Unlock() {}
const bucketSize = 8
type bucket struct {
entries [bucketSize]entry
next *bucket // linked list of buckets
}
type entry struct {
hash uint32 // nonzero => in use
key, value Value
next *entry // insertion order doubly-linked list; may be nil
prevLink **entry // address of link to this entry (perhaps &head)
}
func (ht *hashtable) init(size int) {
if size < 0 {
panic("size < 0")
}
nb := 1
for overloaded(size, nb) {
nb = nb << 1
}
if nb < 2 {
ht.table = ht.bucket0[:1]
} else {
ht.table = make([]bucket, nb)
}
ht.tailLink = &ht.head
}
func (ht *hashtable) freeze() {
if !ht.frozen {
ht.frozen = true
for e := ht.head; e != nil; e = e.next {
e.key.Freeze()
e.value.Freeze()
}
}
}
func (ht *hashtable) insert(k, v Value) error {
if err := ht.checkMutable("insert into"); err != nil {
return err
}
if ht.table == nil {
ht.init(1)
}
h, err := k.Hash()
if err != nil {
return err
}
if h == 0 {
h = 1 // zero is reserved
}
retry:
var insert *entry
// Inspect each bucket in the bucket list.
p := &ht.table[h&(uint32(len(ht.table)-1))]
for {
for i := range p.entries {
e := &p.entries[i]
if e.hash != h {
if e.hash == 0 {
// Found empty entry; make a note.
insert = e
}
continue
}
if eq, err := Equal(k, e.key); err != nil {
return err // e.g. excessively recursive tuple
} else if !eq {
continue
}
// Key already present; update value.
e.value = v
return nil
}
if p.next == nil {
break
}
p = p.next
}
// Key not found. p points to the last bucket.
// Does the number of elements exceed the buckets' load factor?
if overloaded(int(ht.len), len(ht.table)) {
ht.grow()
goto retry
}
if insert == nil {
// No space in existing buckets. Add a new one to the bucket list.
b := new(bucket)
p.next = b
insert = &b.entries[0]
}
// Insert key/value pair.
insert.hash = h
insert.key = k
insert.value = v
// Append entry to doubly-linked list.
insert.prevLink = ht.tailLink
*ht.tailLink = insert
ht.tailLink = &insert.next
ht.len++
return nil
}
func overloaded(elems, buckets int) bool {
const loadFactor = 6.5 // just a guess
return elems >= bucketSize && float64(elems) >= loadFactor*float64(buckets)
}
func (ht *hashtable) grow() {
// Double the number of buckets and rehash.
//
// Even though this makes reentrant calls to ht.insert,
// calls Equals unnecessarily (since there can't be duplicate keys),
// and recomputes the hash unnecessarily, the gains from
// avoiding these steps were found to be too small to justify
// the extra logic: -2% on hashtable benchmark.
ht.table = make([]bucket, len(ht.table)<<1)
oldhead := ht.head
ht.head = nil
ht.tailLink = &ht.head
ht.len = 0
for e := oldhead; e != nil; e = e.next {
ht.insert(e.key, e.value)
}
ht.bucket0[0] = bucket{} // clear out unused initial bucket
}
func (ht *hashtable) lookup(k Value) (v Value, found bool, err error) {
h, err := k.Hash()
if err != nil {
return nil, false, err // unhashable
}
if h == 0 {
h = 1 // zero is reserved
}
if ht.table == nil {
return None, false, nil // empty
}
// Inspect each bucket in the bucket list.
for p := &ht.table[h&(uint32(len(ht.table)-1))]; p != nil; p = p.next {
for i := range p.entries {
e := &p.entries[i]
if e.hash == h {
if eq, err := Equal(k, e.key); err != nil {
return nil, false, err // e.g. excessively recursive tuple
} else if eq {
return e.value, true, nil // found
}
}
}
}
return None, false, nil // not found
}
// count returns the number of distinct elements of iter that are elements of ht.
func (ht *hashtable) count(iter Iterator) (int, error) {
if ht.table == nil {
return 0, nil // empty
}
var k Value
count := 0
// Use a bitset per table entry to record seen elements of ht.
// Elements are identified by their bucket number and index within the bucket.
// Each bitset gets one word initially, but may grow.
storage := make([]big.Word, len(ht.table))
bitsets := make([]big.Int, len(ht.table))
for i := range bitsets {
bitsets[i].SetBits(storage[i : i+1 : i+1])
}
for iter.Next(&k) && count != int(ht.len) {
h, err := k.Hash()
if err != nil {
return 0, err // unhashable
}
if h == 0 {
h = 1 // zero is reserved
}
// Inspect each bucket in the bucket list.
bucketId := h & (uint32(len(ht.table) - 1))
i := 0
for p := &ht.table[bucketId]; p != nil; p = p.next {
for j := range p.entries {
e := &p.entries[j]
if e.hash == h {
if eq, err := Equal(k, e.key); err != nil {
return 0, err
} else if eq {
bitIndex := i<<3 + j
if bitsets[bucketId].Bit(bitIndex) == 0 {
bitsets[bucketId].SetBit(&bitsets[bucketId], bitIndex, 1)
count++
}
}
}
}
i++
}
}
return count, nil
}
// Items returns all the items in the map (as key/value pairs) in insertion order.
func (ht *hashtable) items() []Tuple {
items := make([]Tuple, 0, ht.len)
array := make([]Value, ht.len*2) // allocate a single backing array
for e := ht.head; e != nil; e = e.next {
pair := Tuple(array[:2:2])
array = array[2:]
pair[0] = e.key
pair[1] = e.value
items = append(items, pair)
}
return items
}
func (ht *hashtable) first() (Value, bool) {
if ht.head != nil {
return ht.head.key, true
}
return None, false
}
func (ht *hashtable) keys() []Value {
keys := make([]Value, 0, ht.len)
for e := ht.head; e != nil; e = e.next {
keys = append(keys, e.key)
}
return keys
}
func (ht *hashtable) delete(k Value) (v Value, found bool, err error) {
if err := ht.checkMutable("delete from"); err != nil {
return nil, false, err
}
if ht.table == nil {
return None, false, nil // empty
}
h, err := k.Hash()
if err != nil {
return nil, false, err // unhashable
}
if h == 0 {
h = 1 // zero is reserved
}
// Inspect each bucket in the bucket list.
for p := &ht.table[h&(uint32(len(ht.table)-1))]; p != nil; p = p.next {
for i := range p.entries {
e := &p.entries[i]
if e.hash == h {
if eq, err := Equal(k, e.key); err != nil {
return nil, false, err
} else if eq {
// Remove e from doubly-linked list.
*e.prevLink = e.next
if e.next == nil {
ht.tailLink = e.prevLink // deletion of last entry
} else {
e.next.prevLink = e.prevLink
}
v := e.value
*e = entry{}
ht.len--
return v, true, nil // found
}
}
}
}
// TODO(adonovan): opt: remove completely empty bucket from bucket list.
return None, false, nil // not found
}
// checkMutable reports an error if the hash table should not be mutated.
// verb+" dict" should describe the operation.
func (ht *hashtable) checkMutable(verb string) error {
if ht.frozen {
return fmt.Errorf("cannot %s frozen hash table", verb)
}
if ht.itercount > 0 {
return fmt.Errorf("cannot %s hash table during iteration", verb)
}
return nil
}
func (ht *hashtable) clear() error {
if err := ht.checkMutable("clear"); err != nil {
return err
}
if ht.table != nil {
for i := range ht.table {
ht.table[i] = bucket{}
}
}
ht.head = nil
ht.tailLink = &ht.head
ht.len = 0
return nil
}
func (ht *hashtable) addAll(other *hashtable) error {
for e := other.head; e != nil; e = e.next {
if err := ht.insert(e.key, e.value); err != nil {
return err
}
}
return nil
}
// dump is provided as an aid to debugging.
func (ht *hashtable) dump() {
fmt.Printf("hashtable %p len=%d head=%p tailLink=%p",
ht, ht.len, ht.head, ht.tailLink)
if ht.tailLink != nil {
fmt.Printf(" *tailLink=%p", *ht.tailLink)
}
fmt.Println()
for j := range ht.table {
fmt.Printf("bucket chain %d\n", j)
for p := &ht.table[j]; p != nil; p = p.next {
fmt.Printf("bucket %p\n", p)
for i := range p.entries {
e := &p.entries[i]
fmt.Printf("\tentry %d @ %p hash=%d key=%v value=%v\n",
i, e, e.hash, e.key, e.value)
fmt.Printf("\t\tnext=%p &next=%p prev=%p",
e.next, &e.next, e.prevLink)
if e.prevLink != nil {
fmt.Printf(" *prev=%p", *e.prevLink)
}
fmt.Println()
}
}
}
}
func (ht *hashtable) iterate() *keyIterator {
if !ht.frozen {
ht.itercount++
}
return &keyIterator{ht: ht, e: ht.head}
}
type keyIterator struct {
ht *hashtable
e *entry
}
func (it *keyIterator) Next(k *Value) bool {
if it.e != nil {
*k = it.e.key
it.e = it.e.next
return true
}
return false
}
func (it *keyIterator) Done() {
if !it.ht.frozen {
it.ht.itercount--
}
}
// TODO(adonovan): use go1.19's maphash.String.
// hashString computes the hash of s.
func hashString(s string) uint32 {
if len(s) >= 12 {
// Call the Go runtime's optimized hash implementation,
// which uses the AESENC instruction on amd64 machines.
return uint32(goStringHash(s, 0))
}
return softHashString(s)
}
//go:linkname goStringHash runtime.stringHash
func goStringHash(s string, seed uintptr) uintptr
// softHashString computes the 32-bit FNV-1a hash of s in software.
func softHashString(s string) uint32 {
var h uint32 = 2166136261
for i := 0; i < len(s); i++ {
h ^= uint32(s[i])
h *= 16777619
}
return h
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/hashtable_test.go
+139
View File
@@ -0,0 +1,139 @@
// Copyright 2017 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package starlark
import (
"fmt"
"math/rand"
"sync"
"testing"
)
func TestHashtable(t *testing.T) {
makeTestIntsOnce.Do(makeTestInts)
testHashtable(t, make(map[int]bool))
}
func BenchmarkStringHash(b *testing.B) {
for len := 1; len <= 1024; len *= 2 {
buf := make([]byte, len)
rand.New(rand.NewSource(0)).Read(buf)
s := string(buf)
b.Run(fmt.Sprintf("hard-%d", len), func(b *testing.B) {
for i := 0; i < b.N; i++ {
hashString(s)
}
})
b.Run(fmt.Sprintf("soft-%d", len), func(b *testing.B) {
for i := 0; i < b.N; i++ {
softHashString(s)
}
})
}
}
func BenchmarkHashtable(b *testing.B) {
makeTestIntsOnce.Do(makeTestInts)
b.ResetTimer()
for i := 0; i < b.N; i++ {
testHashtable(b, nil)
}
}
const testIters = 10000
var (
// testInts is a zipf-distributed array of Ints and corresponding ints.
// This removes the cost of generating them on the fly during benchmarking.
// Without this, Zipf and MakeInt dominate CPU and memory costs, respectively.
makeTestIntsOnce sync.Once
testInts [3 * testIters]struct {
Int Int
goInt int
}
)
func makeTestInts() {
zipf := rand.NewZipf(rand.New(rand.NewSource(0)), 1.1, 1.0, 1000.0)
for i := range &testInts {
r := int(zipf.Uint64())
testInts[i].goInt = r
testInts[i].Int = MakeInt(r)
}
}
// testHashtable is both a test and a benchmark of hashtable.
// When sane != nil, it acts as a test against the semantics of Go's map.
func testHashtable(tb testing.TB, sane map[int]bool) {
var i int // index into testInts
var ht hashtable
// Insert 10000 random ints into the map.
for j := 0; j < testIters; j++ {
k := testInts[i]
i++
if err := ht.insert(k.Int, None); err != nil {
tb.Fatal(err)
}
if sane != nil {
sane[k.goInt] = true
}
}
// Do 10000 random lookups in the map.
for j := 0; j < testIters; j++ {
k := testInts[i]
i++
_, found, err := ht.lookup(k.Int)
if err != nil {
tb.Fatal(err)
}
if sane != nil {
_, found2 := sane[k.goInt]
if found != found2 {
tb.Fatal("sanity check failed")
}
}
}
// Do 10000 random deletes from the map.
for j := 0; j < testIters; j++ {
k := testInts[i]
i++
_, found, err := ht.delete(k.Int)
if err != nil {
tb.Fatal(err)
}
if sane != nil {
_, found2 := sane[k.goInt]
if found != found2 {
tb.Fatal("sanity check failed")
}
delete(sane, k.goInt)
}
}
if sane != nil {
if int(ht.len) != len(sane) {
tb.Fatal("sanity check failed")
}
}
}
func TestHashtableCount(t *testing.T) {
const count = 1000
ht := new(hashtable)
for i := 0; i < count; i++ {
ht.insert(MakeInt(i), None)
}
if c, err := ht.count(rangeValue{0, count, 1, count}.Iterate()); err != nil {
t.Error(err)
} else if c != count {
t.Errorf("count doesn't match: expected %d got %d", count, c)
}
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/int.go
+459
View File
@@ -0,0 +1,459 @@
// Copyright 2017 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package starlark
import (
"fmt"
"math"
"math/big"
"reflect"
"strconv"
"go.starlark.net/syntax"
)
// Int is the type of a Starlark int.
//
// The zero value is not a legal value; use MakeInt(0).
type Int struct{ impl intImpl }
// --- high-level accessors ---
// MakeInt returns a Starlark int for the specified signed integer.
func MakeInt(x int) Int { return MakeInt64(int64(x)) }
// MakeInt64 returns a Starlark int for the specified int64.
func MakeInt64(x int64) Int {
if math.MinInt32 <= x && x <= math.MaxInt32 {
return makeSmallInt(x)
}
return makeBigInt(big.NewInt(x))
}
// MakeUint returns a Starlark int for the specified unsigned integer.
func MakeUint(x uint) Int { return MakeUint64(uint64(x)) }
// MakeUint64 returns a Starlark int for the specified uint64.
func MakeUint64(x uint64) Int {
if x <= math.MaxInt32 {
return makeSmallInt(int64(x))
}
return makeBigInt(new(big.Int).SetUint64(x))
}
// MakeBigInt returns a Starlark int for the specified big.Int.
// The new Int value will contain a copy of x. The caller is safe to modify x.
func MakeBigInt(x *big.Int) Int {
if isSmall(x) {
return makeSmallInt(x.Int64())
}
z := new(big.Int).Set(x)
return makeBigInt(z)
}
func isSmall(x *big.Int) bool {
n := x.BitLen()
return n < 32 || n == 32 && x.Int64() == math.MinInt32
}
var (
zero, one = makeSmallInt(0), makeSmallInt(1)
oneBig = big.NewInt(1)
_ HasUnary = Int{}
)
// Unary implements the operations +int, -int, and ~int.
func (i Int) Unary(op syntax.Token) (Value, error) {
switch op {
case syntax.MINUS:
return zero.Sub(i), nil
case syntax.PLUS:
return i, nil
case syntax.TILDE:
return i.Not(), nil
}
return nil, nil
}
// Int64 returns the value as an int64.
// If it is not exactly representable the result is undefined and ok is false.
func (i Int) Int64() (_ int64, ok bool) {
iSmall, iBig := i.get()
if iBig != nil {
x, acc := bigintToInt64(iBig)
if acc != big.Exact {
return // inexact
}
return x, true
}
return iSmall, true
}
// BigInt returns a new big.Int with the same value as the Int.
func (i Int) BigInt() *big.Int {
iSmall, iBig := i.get()
if iBig != nil {
return new(big.Int).Set(iBig)
}
return big.NewInt(iSmall)
}
// bigInt returns the value as a big.Int.
// It differs from BigInt in that this method returns the actual
// reference and any modification will change the state of i.
func (i Int) bigInt() *big.Int {
iSmall, iBig := i.get()
if iBig != nil {
return iBig
}
return big.NewInt(iSmall)
}
// Uint64 returns the value as a uint64.
// If it is not exactly representable the result is undefined and ok is false.
func (i Int) Uint64() (_ uint64, ok bool) {
iSmall, iBig := i.get()
if iBig != nil {
x, acc := bigintToUint64(iBig)
if acc != big.Exact {
return // inexact
}
return x, true
}
if iSmall < 0 {
return // inexact
}
return uint64(iSmall), true
}
// The math/big API should provide this function.
func bigintToInt64(i *big.Int) (int64, big.Accuracy) {
sign := i.Sign()
if sign > 0 {
if i.Cmp(maxint64) > 0 {
return math.MaxInt64, big.Below
}
} else if sign < 0 {
if i.Cmp(minint64) < 0 {
return math.MinInt64, big.Above
}
}
return i.Int64(), big.Exact
}
// The math/big API should provide this function.
func bigintToUint64(i *big.Int) (uint64, big.Accuracy) {
sign := i.Sign()
if sign > 0 {
if i.BitLen() > 64 {
return math.MaxUint64, big.Below
}
} else if sign < 0 {
return 0, big.Above
}
return i.Uint64(), big.Exact
}
var (
minint64 = new(big.Int).SetInt64(math.MinInt64)
maxint64 = new(big.Int).SetInt64(math.MaxInt64)
)
func (i Int) Format(s fmt.State, ch rune) {
iSmall, iBig := i.get()
if iBig != nil {
iBig.Format(s, ch)
return
}
big.NewInt(iSmall).Format(s, ch)
}
func (i Int) String() string {
iSmall, iBig := i.get()
if iBig != nil {
return iBig.Text(10)
}
return strconv.FormatInt(iSmall, 10)
}
func (i Int) Type() string { return "int" }
func (i Int) Freeze() {} // immutable
func (i Int) Truth() Bool { return i.Sign() != 0 }
func (i Int) Hash() (uint32, error) {
iSmall, iBig := i.get()
var lo big.Word
if iBig != nil {
lo = iBig.Bits()[0]
} else {
lo = big.Word(iSmall)
}
return 12582917 * uint32(lo+3), nil
}
// Cmp implements comparison of two Int values.
// Required by the TotallyOrdered interface.
func (i Int) Cmp(v Value, depth int) (int, error) {
j := v.(Int)
iSmall, iBig := i.get()
jSmall, jBig := j.get()
if iBig != nil || jBig != nil {
return i.bigInt().Cmp(j.bigInt()), nil
}
return signum64(iSmall - jSmall), nil // safe: int32 operands
}
// Float returns the float value nearest i.
func (i Int) Float() Float {
iSmall, iBig := i.get()
if iBig != nil {
// Fast path for hardware int-to-float conversions.
if iBig.IsUint64() {
return Float(iBig.Uint64())
} else if iBig.IsInt64() {
return Float(iBig.Int64())
} else {
// Fast path for very big ints.
const maxFiniteLen = 1023 + 1 // max exponent value + implicit mantissa bit
if iBig.BitLen() > maxFiniteLen {
return Float(math.Inf(iBig.Sign()))
}
}
f, _ := new(big.Float).SetInt(iBig).Float64()
return Float(f)
}
return Float(iSmall)
}
// finiteFloat returns the finite float value nearest i,
// or an error if the magnitude is too large.
func (i Int) finiteFloat() (Float, error) {
f := i.Float()
if math.IsInf(float64(f), 0) {
return 0, fmt.Errorf("int too large to convert to float")
}
return f, nil
}
func (x Int) Sign() int {
xSmall, xBig := x.get()
if xBig != nil {
return xBig.Sign()
}
return signum64(xSmall)
}
func (x Int) Add(y Int) Int {
xSmall, xBig := x.get()
ySmall, yBig := y.get()
if xBig != nil || yBig != nil {
return MakeBigInt(new(big.Int).Add(x.bigInt(), y.bigInt()))
}
return MakeInt64(xSmall + ySmall)
}
func (x Int) Sub(y Int) Int {
xSmall, xBig := x.get()
ySmall, yBig := y.get()
if xBig != nil || yBig != nil {
return MakeBigInt(new(big.Int).Sub(x.bigInt(), y.bigInt()))
}
return MakeInt64(xSmall - ySmall)
}
func (x Int) Mul(y Int) Int {
xSmall, xBig := x.get()
ySmall, yBig := y.get()
if xBig != nil || yBig != nil {
return MakeBigInt(new(big.Int).Mul(x.bigInt(), y.bigInt()))
}
return MakeInt64(xSmall * ySmall)
}
func (x Int) Or(y Int) Int {
xSmall, xBig := x.get()
ySmall, yBig := y.get()
if xBig != nil || yBig != nil {
return MakeBigInt(new(big.Int).Or(x.bigInt(), y.bigInt()))
}
return makeSmallInt(xSmall | ySmall)
}
func (x Int) And(y Int) Int {
xSmall, xBig := x.get()
ySmall, yBig := y.get()
if xBig != nil || yBig != nil {
return MakeBigInt(new(big.Int).And(x.bigInt(), y.bigInt()))
}
return makeSmallInt(xSmall & ySmall)
}
func (x Int) Xor(y Int) Int {
xSmall, xBig := x.get()
ySmall, yBig := y.get()
if xBig != nil || yBig != nil {
return MakeBigInt(new(big.Int).Xor(x.bigInt(), y.bigInt()))
}
return makeSmallInt(xSmall ^ ySmall)
}
func (x Int) Not() Int {
xSmall, xBig := x.get()
if xBig != nil {
return MakeBigInt(new(big.Int).Not(xBig))
}
return makeSmallInt(^xSmall)
}
func (x Int) Lsh(y uint) Int { return MakeBigInt(new(big.Int).Lsh(x.bigInt(), y)) }
func (x Int) Rsh(y uint) Int { return MakeBigInt(new(big.Int).Rsh(x.bigInt(), y)) }
// Precondition: y is nonzero.
func (x Int) Div(y Int) Int {
xSmall, xBig := x.get()
ySmall, yBig := y.get()
// http://python-history.blogspot.com/2010/08/why-pythons-integer-division-floors.html
if xBig != nil || yBig != nil {
xb, yb := x.bigInt(), y.bigInt()
var quo, rem big.Int
quo.QuoRem(xb, yb, &rem)
if (xb.Sign() < 0) != (yb.Sign() < 0) && rem.Sign() != 0 {
quo.Sub(&quo, oneBig)
}
return MakeBigInt(&quo)
}
quo := xSmall / ySmall
rem := xSmall % ySmall
if (xSmall < 0) != (ySmall < 0) && rem != 0 {
quo -= 1
}
return MakeInt64(quo)
}
// Precondition: y is nonzero.
func (x Int) Mod(y Int) Int {
xSmall, xBig := x.get()
ySmall, yBig := y.get()
if xBig != nil || yBig != nil {
xb, yb := x.bigInt(), y.bigInt()
var quo, rem big.Int
quo.QuoRem(xb, yb, &rem)
if (xb.Sign() < 0) != (yb.Sign() < 0) && rem.Sign() != 0 {
rem.Add(&rem, yb)
}
return MakeBigInt(&rem)
}
rem := xSmall % ySmall
if (xSmall < 0) != (ySmall < 0) && rem != 0 {
rem += ySmall
}
return makeSmallInt(rem)
}
func (i Int) rational() *big.Rat {
iSmall, iBig := i.get()
if iBig != nil {
return new(big.Rat).SetInt(iBig)
}
return new(big.Rat).SetInt64(iSmall)
}
// AsInt32 returns the value of x if is representable as an int32.
func AsInt32(x Value) (int, error) {
i, ok := x.(Int)
if !ok {
return 0, fmt.Errorf("got %s, want int", x.Type())
}
iSmall, iBig := i.get()
if iBig != nil {
return 0, fmt.Errorf("%s out of range", i)
}
return int(iSmall), nil
}
// AsInt sets *ptr to the value of Starlark int x, if it is exactly representable,
// otherwise it returns an error.
// The type of ptr must be one of the pointer types *int, *int8, *int16, *int32, or *int64,
// or one of their unsigned counterparts including *uintptr.
func AsInt(x Value, ptr interface{}) error {
xint, ok := x.(Int)
if !ok {
return fmt.Errorf("got %s, want int", x.Type())
}
bits := reflect.TypeOf(ptr).Elem().Size() * 8
switch ptr.(type) {
case *int, *int8, *int16, *int32, *int64:
i, ok := xint.Int64()
if !ok || bits < 64 && !(-1<<(bits-1) <= i && i < 1<<(bits-1)) {
return fmt.Errorf("%s out of range (want value in signed %d-bit range)", xint, bits)
}
switch ptr := ptr.(type) {
case *int:
*ptr = int(i)
case *int8:
*ptr = int8(i)
case *int16:
*ptr = int16(i)
case *int32:
*ptr = int32(i)
case *int64:
*ptr = int64(i)
}
case *uint, *uint8, *uint16, *uint32, *uint64, *uintptr:
i, ok := xint.Uint64()
if !ok || bits < 64 && i >= 1<<bits {
return fmt.Errorf("%s out of range (want value in unsigned %d-bit range)", xint, bits)
}
switch ptr := ptr.(type) {
case *uint:
*ptr = uint(i)
case *uint8:
*ptr = uint8(i)
case *uint16:
*ptr = uint16(i)
case *uint32:
*ptr = uint32(i)
case *uint64:
*ptr = uint64(i)
case *uintptr:
*ptr = uintptr(i)
}
default:
panic(fmt.Sprintf("invalid argument type: %T", ptr))
}
return nil
}
// NumberToInt converts a number x to an integer value.
// An int is returned unchanged, a float is truncated towards zero.
// NumberToInt reports an error for all other values.
func NumberToInt(x Value) (Int, error) {
switch x := x.(type) {
case Int:
return x, nil
case Float:
f := float64(x)
if math.IsInf(f, 0) {
return zero, fmt.Errorf("cannot convert float infinity to integer")
} else if math.IsNaN(f) {
return zero, fmt.Errorf("cannot convert float NaN to integer")
}
return finiteFloatToInt(x), nil
}
return zero, fmt.Errorf("cannot convert %s to int", x.Type())
}
// finiteFloatToInt converts f to an Int, truncating towards zero.
// f must be finite.
func finiteFloatToInt(f Float) Int {
// We avoid '<= MaxInt64' so that both constants are exactly representable as floats.
// See https://github.com/google/starlark-go/issues/375.
if math.MinInt64 <= f && f < math.MaxInt64+1 {
// small values
return MakeInt64(int64(f))
}
rat := f.rational()
if rat == nil {
panic(f) // non-finite
}
return MakeBigInt(new(big.Int).Div(rat.Num(), rat.Denom()))
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/int_generic.go
+33
View File
@@ -0,0 +1,33 @@
//go:build (!linux && !darwin && !dragonfly && !freebsd && !netbsd && !solaris) || (!amd64 && !arm64 && !mips64x && !ppc64 && !ppc64le && !loong64 && !s390x)
package starlark
// generic Int implementation as a union
import "math/big"
type intImpl struct {
// We use only the signed 32-bit range of small to ensure
// that small+small and small*small do not overflow.
small_ int64 // minint32 <= small <= maxint32
big_ *big.Int // big != nil <=> value is not representable as int32
}
// --- low-level accessors ---
// get returns the small and big components of the Int.
// small is defined only if big is nil.
// small is sign-extended to 64 bits for ease of subsequent arithmetic.
func (i Int) get() (small int64, big *big.Int) {
return i.impl.small_, i.impl.big_
}
// Precondition: math.MinInt32 <= x && x <= math.MaxInt32
func makeSmallInt(x int64) Int {
return Int{intImpl{small_: x}}
}
// Precondition: x cannot be represented as int32.
func makeBigInt(x *big.Int) Int {
return Int{intImpl{big_: x}}
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/int_posix64.go
+89
View File
@@ -0,0 +1,89 @@
//go:build (linux || darwin || dragonfly || freebsd || netbsd || solaris) && (amd64 || arm64 || mips64x || ppc64 || ppc64le || loong64 || s390x)
package starlark
// This file defines an optimized Int implementation for 64-bit machines
// running POSIX. It reserves a 4GB portion of the address space using
// mmap and represents int32 values as addresses within that range. This
// disambiguates int32 values from *big.Int pointers, letting all Int
// values be represented as an unsafe.Pointer, so that Int-to-Value
// interface conversion need not allocate.
// Although iOS (which, like macOS, appears as darwin/arm64) is
// POSIX-compliant, it limits each process to about 700MB of virtual
// address space, which defeats the optimization. Similarly,
// OpenBSD's default ulimit for virtual memory is a measly GB or so.
// On both those platforms the attempted optimization will fail and
// fall back to the slow implementation.
// An alternative approach to this optimization would be to embed the
// int32 values in pointers using odd values, which can be distinguished
// from (even) *big.Int pointers. However, the Go runtime does not allow
// user programs to manufacture pointers to arbitrary locations such as
// within the zero page, or non-span, non-mmap, non-stack locations,
// and it may panic if it encounters them; see Issue #382.
import (
"log"
"math"
"math/big"
"unsafe"
"golang.org/x/sys/unix"
)
// intImpl represents a union of (int32, *big.Int) in a single pointer,
// so that Int-to-Value conversions need not allocate.
//
// The pointer is either a *big.Int, if the value is big, or a pointer into a
// reserved portion of the address space (smallints), if the value is small
// and the address space allocation succeeded.
//
// See int_generic.go for the basic representation concepts.
type intImpl unsafe.Pointer
// get returns the (small, big) arms of the union.
func (i Int) get() (int64, *big.Int) {
if smallints == 0 {
// optimization disabled
if x := (*big.Int)(i.impl); isSmall(x) {
return x.Int64(), nil
} else {
return 0, x
}
}
if ptr := uintptr(i.impl); ptr >= smallints && ptr < smallints+1<<32 {
return math.MinInt32 + int64(ptr-smallints), nil
}
return 0, (*big.Int)(i.impl)
}
// Precondition: math.MinInt32 <= x && x <= math.MaxInt32
func makeSmallInt(x int64) Int {
if smallints == 0 {
// optimization disabled
return Int{intImpl(big.NewInt(x))}
}
return Int{intImpl(uintptr(x-math.MinInt32) + smallints)}
}
// Precondition: x cannot be represented as int32.
func makeBigInt(x *big.Int) Int { return Int{intImpl(x)} }
// smallints is the base address of a 2^32 byte memory region.
// Pointers to addresses in this region represent int32 values.
// We assume smallints is not at the very top of the address space.
//
// Zero means the optimization is disabled and all Ints allocate a big.Int.
var smallints = reserveAddresses(1 << 32)
func reserveAddresses(len int) uintptr {
b, err := unix.Mmap(-1, 0, len, unix.PROT_READ, unix.MAP_PRIVATE|unix.MAP_ANON)
if err != nil {
log.Printf("Starlark failed to allocate 4GB address space: %v. Integer performance may suffer.", err)
return 0 // optimization disabled
}
return uintptr(unsafe.Pointer(&b[0]))
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/int_test.go
+157
View File
@@ -0,0 +1,157 @@
// Copyright 2017 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package starlark
import (
"flag"
"fmt"
"log"
"math"
"math/big"
"os"
"os/exec"
"runtime"
"strings"
"testing"
)
// TestIntOpts exercises integer arithmetic, especially at the boundaries.
func TestIntOpts(t *testing.T) {
f := MakeInt64
left, right := big.NewInt(math.MinInt32), big.NewInt(math.MaxInt32)
for i, test := range []struct {
val Int
want string
}{
// Add
{f(math.MaxInt32).Add(f(1)), "80000000"},
{f(math.MinInt32).Add(f(-1)), "-80000001"},
// Mul
{f(math.MaxInt32).Mul(f(math.MaxInt32)), "3fffffff00000001"},
{f(math.MinInt32).Mul(f(math.MinInt32)), "4000000000000000"},
{f(math.MaxUint32).Mul(f(math.MaxUint32)), "fffffffe00000001"},
{f(math.MinInt32).Mul(f(-1)), "80000000"},
// Div
{f(math.MinInt32).Div(f(-1)), "80000000"},
{f(1 << 31).Div(f(2)), "40000000"},
// And
{f(math.MaxInt32).And(f(math.MaxInt32)), "7fffffff"},
{f(math.MinInt32).And(f(math.MinInt32)), "-80000000"},
{f(1 << 33).And(f(1 << 32)), "0"},
// Mod
{f(1 << 32).Mod(f(2)), "0"},
// Or
{f(1 << 32).Or(f(0)), "100000000"},
{f(math.MaxInt32).Or(f(0)), "7fffffff"},
{f(math.MaxUint32).Or(f(0)), "ffffffff"},
{f(math.MinInt32).Or(f(math.MinInt32)), "-80000000"},
// Xor
{f(math.MinInt32).Xor(f(-1)), "7fffffff"},
// Not
{f(math.MinInt32).Not(), "7fffffff"},
{f(math.MaxInt32).Not(), "-80000000"},
// Shift
{f(1).Lsh(31), "80000000"},
{f(1).Lsh(32), "100000000"},
{f(math.MaxInt32 + 1).Rsh(1), "40000000"},
{f(math.MinInt32 * 2).Rsh(1), "-80000000"},
} {
if got := fmt.Sprintf("%x", test.val); got != test.want {
t.Errorf("%d equals %s, want %s", i, got, test.want)
}
small, big := test.val.get()
if small < math.MinInt32 || math.MaxInt32 < small {
t.Errorf("expected big, %d %s", i, test.val)
}
if big == nil {
continue
}
if small != 0 {
t.Errorf("expected 0 small, %d %s with %d", i, test.val, small)
}
if big.Cmp(left) >= 0 && big.Cmp(right) <= 0 {
t.Errorf("expected small, %d %s", i, test.val)
}
}
}
func TestImmutabilityMakeBigInt(t *testing.T) {
// use max int64 for the test
expect := int64(^uint64(0) >> 1)
mutint := big.NewInt(expect)
value := MakeBigInt(mutint)
mutint.Set(big.NewInt(1))
got, _ := value.Int64()
if got != expect {
t.Errorf("expected %d, got %d", expect, got)
}
}
func TestImmutabilityBigInt(t *testing.T) {
// use 1 and max int64 for the test
for _, expect := range []int64{1, int64(^uint64(0) >> 1)} {
value := MakeBigInt(big.NewInt(expect))
bigint := value.BigInt()
bigint.Set(big.NewInt(2))
got, _ := value.Int64()
if got != expect {
t.Errorf("expected %d, got %d", expect, got)
}
}
}
// TestIntFallback creates a small Int value in a child process with
// limited address space to ensure that it still works, but prints a warning.
func TestIntFallback(t *testing.T) {
if runtime.GOOS != "linux" {
t.Skipf("test disabled on this platform (requires ulimit -v)")
}
exe, err := os.Executable()
if err != nil {
t.Fatalf("can't find file name of executable: %v", err)
}
// ulimit -v limits the address space in KB. Not portable.
// 4GB is enough for the Go runtime but not for the optimization.
cmd := exec.Command("/bin/sh", "-c", fmt.Sprintf("ulimit -v 4000000 && %q --entry=intfallback", exe))
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("intfallback subcommand failed: %v\n%s", err, out)
}
// Check the warning was printed.
if !strings.Contains(string(out), "Integer performance may suffer") {
t.Errorf("expected warning was not printed. Output=<<%s>>", out)
}
}
// intfallback is called in a child process with limited address space.
func intfallback() {
const want = 123
if got, _ := MakeBigInt(big.NewInt(want)).Int64(); got != want {
log.Fatalf("intfallback: got %d, want %d", got, want)
}
}
// The --entry flag invokes an alternate entry point, for use in subprocess tests.
var testEntry = flag.String("entry", "", "child process entry-point")
func TestMain(m *testing.M) {
// In some build systems, notably Blaze, flag.Parse is called before TestMain,
// in violation of the TestMain contract, making this second call a no-op.
flag.Parse()
switch *testEntry {
case "":
os.Exit(m.Run()) // normal case
case "intfallback":
intfallback()
default:
log.Fatalf("unknown entry point: %s", *testEntry)
}
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/interp.go
+704
View File
@@ -0,0 +1,704 @@
package starlark
// This file defines the bytecode interpreter.
import (
"fmt"
"os"
"sync/atomic"
"unsafe"
"go.starlark.net/internal/compile"
"go.starlark.net/internal/spell"
"go.starlark.net/syntax"
)
const vmdebug = false // TODO(adonovan): use a bitfield of specific kinds of error.
// TODO(adonovan):
// - optimize position table.
// - opt: record MaxIterStack during compilation and preallocate the stack.
func (fn *Function) CallInternal(thread *Thread, args Tuple, kwargs []Tuple) (Value, error) {
// Postcondition: args is not mutated. This is stricter than required by Callable,
// but allows CALL to avoid a copy.
f := fn.funcode
if !f.Prog.Recursion {
// detect recursion
for _, fr := range thread.stack[:len(thread.stack)-1] {
// We look for the same function code,
// not function value, otherwise the user could
// defeat the check by writing the Y combinator.
if frfn, ok := fr.Callable().(*Function); ok && frfn.funcode == f {
return nil, fmt.Errorf("function %s called recursively", fn.Name())
}
}
}
fr := thread.frameAt(0)
// Allocate space for stack and locals.
// Logically these do not escape from this frame
// (See https://github.com/golang/go/issues/20533.)
//
// This heap allocation looks expensive, but I was unable to get
// more than 1% real time improvement in a large alloc-heavy
// benchmark (in which this alloc was 8% of alloc-bytes)
// by allocating space for 8 Values in each frame, or
// by allocating stack by slicing an array held by the Thread
// that is expanded in chunks of min(k, nspace), for k=256 or 1024.
nlocals := len(f.Locals)
nspace := nlocals + f.MaxStack
space := make([]Value, nspace)
locals := space[:nlocals:nlocals] // local variables, starting with parameters
stack := space[nlocals:] // operand stack
// Digest arguments and set parameters.
err := setArgs(locals, fn, args, kwargs)
if err != nil {
return nil, thread.evalError(err)
}
fr.locals = locals
if vmdebug {
fmt.Printf("Entering %s @ %s\n", f.Name, f.Position(0))
fmt.Printf("%d stack, %d locals\n", len(stack), len(locals))
defer fmt.Println("Leaving ", f.Name)
}
// Spill indicated locals to cells.
// Each cell is a separate alloc to avoid spurious liveness.
for _, index := range f.Cells {
locals[index] = &cell{locals[index]}
}
// TODO(adonovan): add static check that beneath this point
// - there is exactly one return statement
// - there is no redefinition of 'err'.
var iterstack []Iterator // stack of active iterators
// Use defer so that application panics can pass through
// interpreter without leaving thread in a bad state.
defer func() {
// ITERPOP the rest of the iterator stack.
for _, iter := range iterstack {
iter.Done()
}
fr.locals = nil
}()
sp := 0
var pc uint32
var result Value
code := f.Code
loop:
for {
thread.Steps++
if thread.Steps >= thread.maxSteps {
if thread.OnMaxSteps != nil {
thread.OnMaxSteps(thread)
} else {
thread.Cancel("too many steps")
}
}
if reason := atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(&thread.cancelReason))); reason != nil {
err = fmt.Errorf("Starlark computation cancelled: %s", *(*string)(reason))
break loop
}
fr.pc = pc
op := compile.Opcode(code[pc])
pc++
var arg uint32
if op >= compile.OpcodeArgMin {
// TODO(adonovan): opt: profile this.
// Perhaps compiling big endian would be less work to decode?
for s := uint(0); ; s += 7 {
b := code[pc]
pc++
arg |= uint32(b&0x7f) << s
if b < 0x80 {
break
}
}
}
if vmdebug {
fmt.Fprintln(os.Stderr, stack[:sp]) // very verbose!
compile.PrintOp(f, fr.pc, op, arg)
}
switch op {
case compile.NOP:
// nop
case compile.DUP:
stack[sp] = stack[sp-1]
sp++
case compile.DUP2:
stack[sp] = stack[sp-2]
stack[sp+1] = stack[sp-1]
sp += 2
case compile.POP:
sp--
case compile.EXCH:
stack[sp-2], stack[sp-1] = stack[sp-1], stack[sp-2]
case compile.EQL, compile.NEQ, compile.GT, compile.LT, compile.LE, compile.GE:
op := syntax.Token(op-compile.EQL) + syntax.EQL
y := stack[sp-1]
x := stack[sp-2]
sp -= 2
ok, err2 := Compare(op, x, y)
if err2 != nil {
err = err2
break loop
}
stack[sp] = Bool(ok)
sp++
case compile.PLUS,
compile.MINUS,
compile.STAR,
compile.SLASH,
compile.SLASHSLASH,
compile.PERCENT,
compile.AMP,
compile.PIPE,
compile.CIRCUMFLEX,
compile.LTLT,
compile.GTGT,
compile.IN:
binop := syntax.Token(op-compile.PLUS) + syntax.PLUS
if op == compile.IN {
binop = syntax.IN // IN token is out of order
}
y := stack[sp-1]
x := stack[sp-2]
sp -= 2
z, err2 := Binary(binop, x, y)
if err2 != nil {
err = err2
break loop
}
stack[sp] = z
sp++
case compile.UPLUS, compile.UMINUS, compile.TILDE:
var unop syntax.Token
if op == compile.TILDE {
unop = syntax.TILDE
} else {
unop = syntax.Token(op-compile.UPLUS) + syntax.PLUS
}
x := stack[sp-1]
y, err2 := Unary(unop, x)
if err2 != nil {
err = err2
break loop
}
stack[sp-1] = y
case compile.INPLACE_ADD:
y := stack[sp-1]
x := stack[sp-2]
sp -= 2
// It's possible that y is not Iterable but
// nonetheless defines x+y, in which case we
// should fall back to the general case.
var z Value
if xlist, ok := x.(*List); ok {
if yiter, ok := y.(Iterable); ok {
if err = xlist.checkMutable("apply += to"); err != nil {
break loop
}
listExtend(xlist, yiter)
z = xlist
}
}
if z == nil {
z, err = Binary(syntax.PLUS, x, y)
if err != nil {
break loop
}
}
stack[sp] = z
sp++
case compile.INPLACE_PIPE:
y := stack[sp-1]
x := stack[sp-2]
sp -= 2
// It's possible that y is not Dict but
// nonetheless defines x|y, in which case we
// should fall back to the general case.
var z Value
if xdict, ok := x.(*Dict); ok {
if ydict, ok := y.(*Dict); ok {
if err = xdict.ht.checkMutable("apply |= to"); err != nil {
break loop
}
xdict.ht.addAll(&ydict.ht) // can't fail
z = xdict
}
}
if z == nil {
z, err = Binary(syntax.PIPE, x, y)
if err != nil {
break loop
}
}
stack[sp] = z
sp++
case compile.NONE:
stack[sp] = None
sp++
case compile.TRUE:
stack[sp] = True
sp++
case compile.FALSE:
stack[sp] = False
sp++
case compile.MANDATORY:
stack[sp] = mandatory{}
sp++
case compile.JMP:
pc = arg
case compile.CALL, compile.CALL_VAR, compile.CALL_KW, compile.CALL_VAR_KW:
var kwargs Value
if op == compile.CALL_KW || op == compile.CALL_VAR_KW {
kwargs = stack[sp-1]
sp--
}
var args Value
if op == compile.CALL_VAR || op == compile.CALL_VAR_KW {
args = stack[sp-1]
sp--
}
// named args (pairs)
var kvpairs []Tuple
if nkvpairs := int(arg & 0xff); nkvpairs > 0 {
kvpairs = make([]Tuple, 0, nkvpairs)
kvpairsAlloc := make(Tuple, 2*nkvpairs) // allocate a single backing array
sp -= 2 * nkvpairs
for i := 0; i < nkvpairs; i++ {
pair := kvpairsAlloc[:2:2]
kvpairsAlloc = kvpairsAlloc[2:]
pair[0] = stack[sp+2*i] // name
pair[1] = stack[sp+2*i+1] // value
kvpairs = append(kvpairs, pair)
}
}
if kwargs != nil {
// Add key/value items from **kwargs dictionary.
dict, ok := kwargs.(IterableMapping)
if !ok {
err = fmt.Errorf("argument after ** must be a mapping, not %s", kwargs.Type())
break loop
}
items := dict.Items()
for _, item := range items {
if _, ok := item[0].(String); !ok {
err = fmt.Errorf("keywords must be strings, not %s", item[0].Type())
break loop
}
}
if len(kvpairs) == 0 {
kvpairs = items
} else {
kvpairs = append(kvpairs, items...)
}
}
// positional args
var positional Tuple
if npos := int(arg >> 8); npos > 0 {
positional = stack[sp-npos : sp]
sp -= npos
// Copy positional arguments into a new array,
// unless the callee is another Starlark function,
// in which case it can be trusted not to mutate them.
if _, ok := stack[sp-1].(*Function); !ok || args != nil {
positional = append(Tuple(nil), positional...)
}
}
if args != nil {
// Add elements from *args sequence.
iter := Iterate(args)
if iter == nil {
err = fmt.Errorf("argument after * must be iterable, not %s", args.Type())
break loop
}
var elem Value
for iter.Next(&elem) {
positional = append(positional, elem)
}
iter.Done()
}
function := stack[sp-1]
if vmdebug {
fmt.Printf("VM call %s args=%s kwargs=%s @%s\n",
function, positional, kvpairs, f.Position(fr.pc))
}
thread.endProfSpan()
z, err2 := Call(thread, function, positional, kvpairs)
thread.beginProfSpan()
if err2 != nil {
err = err2
break loop
}
if vmdebug {
fmt.Printf("Resuming %s @ %s\n", f.Name, f.Position(0))
}
stack[sp-1] = z
case compile.ITERPUSH:
x := stack[sp-1]
sp--
iter := Iterate(x)
if iter == nil {
err = fmt.Errorf("%s value is not iterable", x.Type())
break loop
}
iterstack = append(iterstack, iter)
case compile.ITERJMP:
iter := iterstack[len(iterstack)-1]
if iter.Next(&stack[sp]) {
sp++
} else {
pc = arg
}
case compile.ITERPOP:
n := len(iterstack) - 1
iterstack[n].Done()
iterstack = iterstack[:n]
case compile.NOT:
stack[sp-1] = !stack[sp-1].Truth()
case compile.RETURN:
result = stack[sp-1]
break loop
case compile.SETINDEX:
z := stack[sp-1]
y := stack[sp-2]
x := stack[sp-3]
sp -= 3
err = setIndex(x, y, z)
if err != nil {
break loop
}
case compile.INDEX:
y := stack[sp-1]
x := stack[sp-2]
sp -= 2
z, err2 := getIndex(x, y)
if err2 != nil {
err = err2
break loop
}
stack[sp] = z
sp++
case compile.ATTR:
x := stack[sp-1]
name := f.Prog.Names[arg]
y, err2 := getAttr(x, name)
if err2 != nil {
err = err2
break loop
}
stack[sp-1] = y
case compile.SETFIELD:
y := stack[sp-1]
x := stack[sp-2]
sp -= 2
name := f.Prog.Names[arg]
if err2 := setField(x, name, y); err2 != nil {
err = err2
break loop
}
case compile.MAKEDICT:
stack[sp] = new(Dict)
sp++
case compile.SETDICT, compile.SETDICTUNIQ:
dict := stack[sp-3].(*Dict)
k := stack[sp-2]
v := stack[sp-1]
sp -= 3
oldlen := dict.Len()
if err2 := dict.SetKey(k, v); err2 != nil {
err = err2
break loop
}
if op == compile.SETDICTUNIQ && dict.Len() == oldlen {
err = fmt.Errorf("duplicate key: %v", k)
break loop
}
case compile.APPEND:
elem := stack[sp-1]
list := stack[sp-2].(*List)
sp -= 2
list.elems = append(list.elems, elem)
case compile.SLICE:
x := stack[sp-4]
lo := stack[sp-3]
hi := stack[sp-2]
step := stack[sp-1]
sp -= 4
res, err2 := slice(x, lo, hi, step)
if err2 != nil {
err = err2
break loop
}
stack[sp] = res
sp++
case compile.UNPACK:
n := int(arg)
iterable := stack[sp-1]
sp--
iter := Iterate(iterable)
if iter == nil {
err = fmt.Errorf("got %s in sequence assignment", iterable.Type())
break loop
}
i := 0
sp += n
for i < n && iter.Next(&stack[sp-1-i]) {
i++
}
var dummy Value
if iter.Next(&dummy) {
// NB: Len may return -1 here in obscure cases.
err = fmt.Errorf("too many values to unpack (got %d, want %d)", Len(iterable), n)
break loop
}
iter.Done()
if i < n {
err = fmt.Errorf("too few values to unpack (got %d, want %d)", i, n)
break loop
}
case compile.CJMP:
if stack[sp-1].Truth() {
pc = arg
}
sp--
case compile.CONSTANT:
stack[sp] = fn.module.constants[arg]
sp++
case compile.MAKETUPLE:
n := int(arg)
tuple := make(Tuple, n)
sp -= n
copy(tuple, stack[sp:])
stack[sp] = tuple
sp++
case compile.MAKELIST:
n := int(arg)
elems := make([]Value, n)
sp -= n
copy(elems, stack[sp:])
stack[sp] = NewList(elems)
sp++
case compile.MAKEFUNC:
funcode := f.Prog.Functions[arg]
tuple := stack[sp-1].(Tuple)
n := len(tuple) - len(funcode.Freevars)
defaults := tuple[:n:n]
freevars := tuple[n:]
stack[sp-1] = &Function{
funcode: funcode,
module: fn.module,
defaults: defaults,
freevars: freevars,
}
case compile.LOAD:
n := int(arg)
module := string(stack[sp-1].(String))
sp--
if thread.Load == nil {
err = fmt.Errorf("load not implemented by this application")
break loop
}
thread.endProfSpan()
dict, err2 := thread.Load(thread, module)
thread.beginProfSpan()
if err2 != nil {
err = wrappedError{
msg: fmt.Sprintf("cannot load %s: %v", module, err2),
cause: err2,
}
break loop
}
for i := 0; i < n; i++ {
from := string(stack[sp-1-i].(String))
v, ok := dict[from]
if !ok {
err = fmt.Errorf("load: name %s not found in module %s", from, module)
if n := spell.Nearest(from, dict.Keys()); n != "" {
err = fmt.Errorf("%s (did you mean %s?)", err, n)
}
break loop
}
stack[sp-1-i] = v
}
case compile.SETLOCAL:
locals[arg] = stack[sp-1]
sp--
case compile.SETLOCALCELL:
locals[arg].(*cell).v = stack[sp-1]
sp--
case compile.SETGLOBAL:
fn.module.globals[arg] = stack[sp-1]
sp--
case compile.LOCAL:
x := locals[arg]
if x == nil {
err = fmt.Errorf("local variable %s referenced before assignment", f.Locals[arg].Name)
break loop
}
stack[sp] = x
sp++
case compile.FREE:
stack[sp] = fn.freevars[arg]
sp++
case compile.LOCALCELL:
v := locals[arg].(*cell).v
if v == nil {
err = fmt.Errorf("local variable %s referenced before assignment", f.Locals[arg].Name)
break loop
}
stack[sp] = v
sp++
case compile.FREECELL:
v := fn.freevars[arg].(*cell).v
if v == nil {
err = fmt.Errorf("local variable %s referenced before assignment", f.Freevars[arg].Name)
break loop
}
stack[sp] = v
sp++
case compile.GLOBAL:
x := fn.module.globals[arg]
if x == nil {
err = fmt.Errorf("global variable %s referenced before assignment", f.Prog.Globals[arg].Name)
break loop
}
stack[sp] = x
sp++
case compile.PREDECLARED:
name := f.Prog.Names[arg]
x := fn.module.predeclared[name]
if x == nil {
err = fmt.Errorf("internal error: predeclared variable %s is uninitialized", name)
break loop
}
stack[sp] = x
sp++
case compile.UNIVERSAL:
stack[sp] = Universe[f.Prog.Names[arg]]
sp++
default:
err = fmt.Errorf("unimplemented: %s", op)
break loop
}
}
// (deferred cleanup runs here)
return result, err
}
type wrappedError struct {
msg string
cause error
}
func (e wrappedError) Error() string {
return e.msg
}
// Implements the xerrors.Wrapper interface
// https://godoc.org/golang.org/x/xerrors#Wrapper
func (e wrappedError) Unwrap() error {
return e.cause
}
// mandatory is a sentinel value used in a function's defaults tuple
// to indicate that a (keyword-only) parameter is mandatory.
type mandatory struct{}
func (mandatory) String() string { return "mandatory" }
func (mandatory) Type() string { return "mandatory" }
func (mandatory) Freeze() {} // immutable
func (mandatory) Truth() Bool { return False }
func (mandatory) Hash() (uint32, error) { return 0, nil }
// A cell is a box containing a Value.
// Local variables marked as cells hold their value indirectly
// so that they may be shared by outer and inner nested functions.
// Cells are always accessed using indirect {FREE,LOCAL,SETLOCAL}CELL instructions.
// The FreeVars tuple contains only cells.
// The FREE instruction always yields a cell.
type cell struct{ v Value }
func (c *cell) String() string { return "cell" }
func (c *cell) Type() string { return "cell" }
func (c *cell) Freeze() {
if c.v != nil {
c.v.Freeze()
}
}
func (c *cell) Truth() Bool { panic("unreachable") }
func (c *cell) Hash() (uint32, error) { panic("unreachable") }
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/library.go
+2455
View File
File diff suppressed because it is too large Load Diff
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/profile.go
+449
View File
@@ -0,0 +1,449 @@
// Copyright 2019 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package starlark
// This file defines a simple execution-time profiler for Starlark.
// It measures the wall time spent executing Starlark code, and emits a
// gzipped protocol message in pprof format (github.com/google/pprof).
//
// When profiling is enabled, the interpreter calls the profiler to
// indicate the start and end of each "span" or time interval. A leaf
// function (whether Go or Starlark) has a single span. A function that
// calls another function has spans for each interval in which it is the
// top of the stack. (A LOAD instruction also ends a span.)
//
// At the start of a span, the interpreter records the current time in
// the thread's topmost frame. At the end of the span, it obtains the
// time again and subtracts the span start time. The difference is added
// to an accumulator variable in the thread. If the accumulator exceeds
// some fixed quantum (10ms, say), the profiler records the current call
// stack and sends it to the profiler goroutine, along with the number
// of quanta, which are subtracted. For example, if the accumulator
// holds 3ms and then a completed span adds 25ms to it, its value is 28ms,
// which exceeeds 10ms. The profiler records a stack with the value 20ms
// (2 quanta), and the accumulator is left with 8ms.
//
// The profiler goroutine converts the stacks into the pprof format and
// emits a gzip-compressed protocol message to the designated output
// file. We use a hand-written streaming proto encoder to avoid
// dependencies on pprof and proto, and to avoid the need to
// materialize the profile data structure in memory.
//
// A limitation of this profiler is that it measures wall time, which
// does not necessarily correspond to CPU time. A CPU profiler requires
// that only running (not runnable) threads are sampled; this is
// commonly achieved by having the kernel deliver a (PROF) signal to an
// arbitrary running thread, through setitimer(2). The CPU profiler in the
// Go runtime uses this mechanism, but it is not possible for a Go
// application to register a SIGPROF handler, nor is it possible for a
// Go handler for some other signal to read the stack pointer of
// the interrupted thread.
//
// Two caveats:
// (1) it is tempting to send the leaf Frame directly to the profiler
// goroutine instead of making a copy of the stack, since a Frame is a
// spaghetti stack--a linked list. However, as soon as execution
// resumes, the stack's Frame.pc values may be mutated, so Frames are
// not safe to share with the asynchronous profiler goroutine.
// (2) it is tempting to use Callables as keys in a map when tabulating
// the pprof protocols's Function entities. However, we cannot assume
// that Callables are valid map keys, and furthermore we must not
// pin function values in memory indefinitely as this may cause lambda
// values to keep their free variables live much longer than necessary.
// TODO(adonovan):
// - make Start/Stop fully thread-safe.
// - fix the pc hack.
// - experiment with other values of quantum.
import (
"bufio"
"bytes"
"compress/gzip"
"encoding/binary"
"fmt"
"io"
"log"
"reflect"
"sync/atomic"
"time"
"unsafe"
"go.starlark.net/syntax"
)
// StartProfile enables time profiling of all Starlark threads,
// and writes a profile in pprof format to w.
// It must be followed by a call to StopProfiler to stop
// the profiler and finalize the profile.
//
// StartProfile returns an error if profiling was already enabled.
//
// StartProfile must not be called concurrently with Starlark execution.
func StartProfile(w io.Writer) error {
if !atomic.CompareAndSwapUint32(&profiler.on, 0, 1) {
return fmt.Errorf("profiler already running")
}
// TODO(adonovan): make the API fully concurrency-safe.
// The main challenge is racy reads/writes of profiler.events,
// and of send/close races on the channel it refers to.
// It's easy to solve them with a mutex but harder to do
// it efficiently.
profiler.events = make(chan *profEvent, 1)
profiler.done = make(chan error)
go profile(w)
return nil
}
// StopProfile stops the profiler started by a prior call to
// StartProfile and finalizes the profile. It returns an error if the
// profile could not be completed.
//
// StopProfile must not be called concurrently with Starlark execution.
func StopProfile() error {
// Terminate the profiler goroutine and get its result.
close(profiler.events)
err := <-profiler.done
profiler.done = nil
profiler.events = nil
atomic.StoreUint32(&profiler.on, 0)
return err
}
// globals
var profiler struct {
on uint32 // nonzero => profiler running
events chan *profEvent // profile events from interpreter threads
done chan error // indicates profiler goroutine is ready
}
func (thread *Thread) beginProfSpan() {
if profiler.events == nil {
return // profiling not enabled
}
thread.frameAt(0).spanStart = nanotime()
}
// TODO(adonovan): experiment with smaller values,
// which trade space and time for greater precision.
const quantum = 10 * time.Millisecond
func (thread *Thread) endProfSpan() {
if profiler.events == nil {
return // profiling not enabled
}
// Add the span to the thread's accumulator.
thread.proftime += time.Duration(nanotime() - thread.frameAt(0).spanStart)
if thread.proftime < quantum {
return
}
// Only record complete quanta.
n := thread.proftime / quantum
thread.proftime -= n * quantum
// Copy the stack.
// (We can't save thread.frame because its pc will change.)
ev := &profEvent{
thread: thread,
time: n * quantum,
}
ev.stack = ev.stackSpace[:0]
for i := range thread.stack {
fr := thread.frameAt(i)
ev.stack = append(ev.stack, profFrame{
pos: fr.Position(),
fn: fr.Callable(),
pc: fr.pc,
})
}
profiler.events <- ev
}
type profEvent struct {
thread *Thread // currently unused
time time.Duration
stack []profFrame
stackSpace [8]profFrame // initial space for stack
}
type profFrame struct {
fn Callable // don't hold this live for too long (prevents GC of lambdas)
pc uint32 // program counter (Starlark frames only)
pos syntax.Position // position of pc within this frame
}
// profile is the profiler goroutine.
// It runs until StopProfiler is called.
func profile(w io.Writer) {
// Field numbers from pprof protocol.
// See https://github.com/google/pprof/blob/master/proto/profile.proto
const (
Profile_sample_type = 1 // repeated ValueType
Profile_sample = 2 // repeated Sample
Profile_mapping = 3 // repeated Mapping
Profile_location = 4 // repeated Location
Profile_function = 5 // repeated Function
Profile_string_table = 6 // repeated string
Profile_time_nanos = 9 // int64
Profile_duration_nanos = 10 // int64
Profile_period_type = 11 // ValueType
Profile_period = 12 // int64
ValueType_type = 1 // int64
ValueType_unit = 2 // int64
Sample_location_id = 1 // repeated uint64
Sample_value = 2 // repeated int64
Sample_label = 3 // repeated Label
Label_key = 1 // int64
Label_str = 2 // int64
Label_num = 3 // int64
Label_num_unit = 4 // int64
Location_id = 1 // uint64
Location_mapping_id = 2 // uint64
Location_address = 3 // uint64
Location_line = 4 // repeated Line
Line_function_id = 1 // uint64
Line_line = 2 // int64
Function_id = 1 // uint64
Function_name = 2 // int64
Function_system_name = 3 // int64
Function_filename = 4 // int64
Function_start_line = 5 // int64
)
bufw := bufio.NewWriter(w) // write file in 4KB (not 240B flate-sized) chunks
gz := gzip.NewWriter(bufw)
enc := protoEncoder{w: gz}
// strings
stringIndex := make(map[string]int64)
str := func(s string) int64 {
i, ok := stringIndex[s]
if !ok {
i = int64(len(stringIndex))
enc.string(Profile_string_table, s)
stringIndex[s] = i
}
return i
}
str("") // entry 0
// functions
//
// function returns the ID of a Callable for use in Line.FunctionId.
// The ID is the same as the function's logical address,
// which is supplied by the caller to avoid the need to recompute it.
functionId := make(map[uintptr]uint64)
function := func(fn Callable, addr uintptr) uint64 {
id, ok := functionId[addr]
if !ok {
id = uint64(addr)
var pos syntax.Position
if fn, ok := fn.(callableWithPosition); ok {
pos = fn.Position()
}
name := fn.Name()
if name == "<toplevel>" {
name = pos.Filename()
}
nameIndex := str(name)
fun := new(bytes.Buffer)
funenc := protoEncoder{w: fun}
funenc.uint(Function_id, id)
funenc.int(Function_name, nameIndex)
funenc.int(Function_system_name, nameIndex)
funenc.int(Function_filename, str(pos.Filename()))
funenc.int(Function_start_line, int64(pos.Line))
enc.bytes(Profile_function, fun.Bytes())
functionId[addr] = id
}
return id
}
// locations
//
// location returns the ID of the location denoted by fr.
// For Starlark frames, this is the Frame pc.
locationId := make(map[uintptr]uint64)
location := func(fr profFrame) uint64 {
fnAddr := profFuncAddr(fr.fn)
// For Starlark functions, the frame position
// represents the current PC value.
// Mix it into the low bits of the address.
// This is super hacky and may result in collisions
// in large functions or if functions are numerous.
// TODO(adonovan): fix: try making this cleaner by treating
// each bytecode segment as a Profile.Mapping.
pcAddr := fnAddr
if _, ok := fr.fn.(*Function); ok {
pcAddr = (pcAddr << 16) ^ uintptr(fr.pc)
}
id, ok := locationId[pcAddr]
if !ok {
id = uint64(pcAddr)
line := new(bytes.Buffer)
lineenc := protoEncoder{w: line}
lineenc.uint(Line_function_id, function(fr.fn, fnAddr))
lineenc.int(Line_line, int64(fr.pos.Line))
loc := new(bytes.Buffer)
locenc := protoEncoder{w: loc}
locenc.uint(Location_id, id)
locenc.uint(Location_address, uint64(pcAddr))
locenc.bytes(Location_line, line.Bytes())
enc.bytes(Profile_location, loc.Bytes())
locationId[pcAddr] = id
}
return id
}
wallNanos := new(bytes.Buffer)
wnenc := protoEncoder{w: wallNanos}
wnenc.int(ValueType_type, str("wall"))
wnenc.int(ValueType_unit, str("nanoseconds"))
// informational fields of Profile
enc.bytes(Profile_sample_type, wallNanos.Bytes())
enc.int(Profile_period, quantum.Nanoseconds()) // magnitude of sampling period
enc.bytes(Profile_period_type, wallNanos.Bytes()) // dimension and unit of period
enc.int(Profile_time_nanos, time.Now().UnixNano()) // start (real) time of profile
startNano := nanotime()
// Read profile events from the channel
// until it is closed by StopProfiler.
for e := range profiler.events {
sample := new(bytes.Buffer)
sampleenc := protoEncoder{w: sample}
sampleenc.int(Sample_value, e.time.Nanoseconds()) // wall nanoseconds
for _, fr := range e.stack {
sampleenc.uint(Sample_location_id, location(fr))
}
enc.bytes(Profile_sample, sample.Bytes())
}
endNano := nanotime()
enc.int(Profile_duration_nanos, endNano-startNano)
err := gz.Close() // Close reports any prior write error
if flushErr := bufw.Flush(); err == nil {
err = flushErr
}
profiler.done <- err
}
// nanotime returns the time in nanoseconds since epoch.
// It is implemented by runtime.nanotime using the linkname hack;
// runtime.nanotime is defined for all OSs/ARCHS and uses the
// monotonic system clock, which there is no portable way to access.
// Should that function ever go away, these alternatives exist:
//
// // POSIX only. REALTIME not MONOTONIC. 17ns.
// var tv syscall.Timeval
// syscall.Gettimeofday(&tv) // can't fail
// return tv.Nano()
//
// // Portable. REALTIME not MONOTONIC. 46ns.
// return time.Now().Nanoseconds()
//
// // POSIX only. Adds a dependency.
// import "golang.org/x/sys/unix"
// var ts unix.Timespec
// unix.ClockGettime(CLOCK_MONOTONIC, &ts) // can't fail
// return unix.TimespecToNsec(ts)
//
//go:linkname nanotime runtime.nanotime
func nanotime() int64
// profFuncAddr returns the canonical "address"
// of a Callable for use by the profiler.
func profFuncAddr(fn Callable) uintptr {
switch fn := fn.(type) {
case *Builtin:
return reflect.ValueOf(fn.fn).Pointer()
case *Function:
return uintptr(unsafe.Pointer(fn.funcode))
}
// User-defined callable types are typically of
// of kind pointer-to-struct. Handle them specially.
if v := reflect.ValueOf(fn); v.Type().Kind() == reflect.Ptr {
return v.Pointer()
}
// Address zero is reserved by the protocol.
// Use 1 for callables we don't recognize.
log.Printf("Starlark profiler: no address for Callable %T", fn)
return 1
}
// We encode the protocol message by hand to avoid making
// the interpreter depend on both github.com/google/pprof
// and github.com/golang/protobuf.
//
// This also avoids the need to materialize a protocol message object
// tree of unbounded size and serialize it all at the end.
// The pprof format appears to have been designed to
// permit streaming implementations such as this one.
//
// See https://developers.google.com/protocol-buffers/docs/encoding.
type protoEncoder struct {
w io.Writer // *bytes.Buffer or *gzip.Writer
tmp [binary.MaxVarintLen64]byte
}
func (e *protoEncoder) uvarint(x uint64) {
n := binary.PutUvarint(e.tmp[:], x)
e.w.Write(e.tmp[:n])
}
func (e *protoEncoder) tag(field, wire uint) {
e.uvarint(uint64(field<<3 | wire))
}
func (e *protoEncoder) string(field uint, s string) {
e.tag(field, 2) // length-delimited
e.uvarint(uint64(len(s)))
io.WriteString(e.w, s)
}
func (e *protoEncoder) bytes(field uint, b []byte) {
e.tag(field, 2) // length-delimited
e.uvarint(uint64(len(b)))
e.w.Write(b)
}
func (e *protoEncoder) uint(field uint, x uint64) {
e.tag(field, 0) // varint
e.uvarint(x)
}
func (e *protoEncoder) int(field uint, x int64) {
e.tag(field, 0) // varint
e.uvarint(uint64(x))
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/profile_test.go
+81
View File
@@ -0,0 +1,81 @@
// Copyright 2019 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package starlark_test
import (
"bytes"
"fmt"
"os"
"os/exec"
"strings"
"testing"
"go.starlark.net/starlark"
)
// TestProfile is a simple integration test that the profiler
// emits minimally plausible pprof-compatible output.
func TestProfile(t *testing.T) {
prof, err := os.CreateTemp(t.TempDir(), "profile_test")
if err != nil {
t.Fatal(err)
}
defer prof.Close()
if err := starlark.StartProfile(prof); err != nil {
t.Fatal(err)
}
const src = `
def fibonacci(n):
x, y = 1, 1
for i in range(n):
x, y = y, x+y
return y
fibonacci(100000)
`
thread := new(starlark.Thread)
if _, err := starlark.ExecFile(thread, "foo.star", src, nil); err != nil {
_ = starlark.StopProfile()
t.Fatal(err)
}
if err := starlark.StopProfile(); err != nil {
t.Fatal(err)
}
prof.Sync()
cmd := exec.Command("go", "tool", "pprof", "-top", prof.Name())
cmd.Stderr = new(bytes.Buffer)
cmd.Stdout = new(bytes.Buffer)
if err := cmd.Run(); err != nil {
t.Fatalf("pprof failed: %v; output=<<%s>>", err, cmd.Stderr)
}
// Typical output (may vary by go release):
//
// Type: wall
// Time: Apr 4, 2019 at 11:10am (EDT)
// Duration: 251.62ms, Total samples = 250ms (99.36%)
// Showing nodes accounting for 250ms, 100% of 250ms total
// flat flat% sum% cum cum%
// 320ms 100% 100% 320ms 100% fibonacci
// 0 0% 100% 320ms 100% foo.star
//
// We'll assert a few key substrings are present.
got := fmt.Sprint(cmd.Stdout)
for _, want := range []string{
"flat%",
"fibonacci",
"foo.star",
} {
if !strings.Contains(got, want) {
t.Errorf("output did not contain %q", want)
}
}
if t.Failed() {
t.Logf("stderr=%v", cmd.Stderr)
t.Logf("stdout=%v", cmd.Stdout)
}
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/assign.star
Vendored
+354
View File
@@ -0,0 +1,354 @@
# Tests of Starlark assignment.
# This is a "chunked" file: each "---" effectively starts a new file.
# tuple assignment
load("assert.star", "assert")
() = () # empty ok
a, b, c = 1, 2, 3
assert.eq(a, 1)
assert.eq(b, 2)
assert.eq(c, 3)
(d, e, f,) = (1, 2, 3) # trailing comma ok
---
(a, b, c) = 1 ### "got int in sequence assignment"
---
(a, b) = () ### "too few values to unpack"
---
(a, b) = (1,) ### "too few values to unpack"
---
(a, b, c) = (1, 2) ### "too few values to unpack"
---
(a, b) = (1, 2, 3) ### "too many values to unpack"
---
() = 1 ### "got int in sequence assignment"
---
() = (1,) ### "too many values to unpack"
---
() = (1, 2) ### "too many values to unpack"
---
# list assignment
load("assert.star", "assert")
[] = [] # empty ok
[a, b, c] = [1, 2, 3]
assert.eq(a, 1)
assert.eq(b, 2)
assert.eq(c, 3)
[d, e, f,] = [1, 2, 3] # trailing comma ok
---
[a, b, c] = 1 ### "got int in sequence assignment"
---
[a, b] = [] ### "too few values to unpack"
---
[a, b] = [1] ### "too few values to unpack"
---
[a, b, c] = [1, 2] ### "too few values to unpack"
---
[a, b] = [1, 2, 3] ### "too many values to unpack"
---
[] = 1 ### "got int in sequence assignment"
---
[] = [1] ### "too many values to unpack"
---
[] = [1, 2] ### "too many values to unpack"
---
# list-tuple assignment
load("assert.star", "assert")
# empty ok
[] = ()
() = []
[a, b, c] = (1, 2, 3)
assert.eq(a, 1)
assert.eq(b, 2)
assert.eq(c, 3)
[a2, b2, c2] = 1, 2, 3 # bare tuple ok
(d, e, f) = [1, 2, 3]
assert.eq(d, 1)
assert.eq(e, 2)
assert.eq(f, 3)
[g, h, (i, j)] = (1, 2, [3, 4])
assert.eq(g, 1)
assert.eq(h, 2)
assert.eq(i, 3)
assert.eq(j, 4)
(k, l, [m, n]) = [1, 2, (3, 4)]
assert.eq(k, 1)
assert.eq(l, 2)
assert.eq(m, 3)
assert.eq(n, 4)
---
# misc assignment
load("assert.star", "assert")
def assignment():
a = [1, 2, 3]
a[1] = 5
assert.eq(a, [1, 5, 3])
a[-2] = 2
assert.eq(a, [1, 2, 3])
assert.eq("%d %d" % (5, 7), "5 7")
x={}
x[1] = 2
x[1] += 3
assert.eq(x[1], 5)
def f12(): x[(1, "abc", {})] = 1
assert.fails(f12, "unhashable type: dict")
assignment()
---
# augmented assignment
load("assert.star", "assert")
def f():
x = 1
x += 1
assert.eq(x, 2)
x *= 3
assert.eq(x, 6)
f()
---
# effects of evaluating LHS occur only once
load("assert.star", "assert")
count = [0] # count[0] is the number of calls to f
def f():
count[0] += 1
return count[0]
x = [1, 2, 3]
x[f()] += 1
assert.eq(x, [1, 3, 3]) # sole call to f returned 1
assert.eq(count[0], 1) # f was called only once
---
# Order of evaluation.
load("assert.star", "assert")
calls = []
def f(name, result):
calls.append(name)
return result
# The right side is evaluated before the left in an ordinary assignment.
calls.clear()
f("array", [0])[f("index", 0)] = f("rhs", 0)
assert.eq(calls, ["rhs", "array", "index"])
calls.clear()
f("lhs1", [0])[0], f("lhs2", [0])[0] = f("rhs1", 0), f("rhs2", 0)
assert.eq(calls, ["rhs1", "rhs2", "lhs1", "lhs2"])
# Left side is evaluated first (and only once) in an augmented assignment.
calls.clear()
f("array", [0])[f("index", 0)] += f("addend", 1)
assert.eq(calls, ["array", "index", "addend"])
---
# global referenced before assignment
def f():
return g ### "global variable g referenced before assignment"
f()
g = 1
---
# Free variables are captured by reference, so this is ok.
load("assert.star", "assert")
def f():
def g():
return outer
outer = 1
return g()
assert.eq(f(), 1)
---
load("assert.star", "assert")
printok = [False]
# This program should resolve successfully but fail dynamically.
# However, the Java implementation currently reports the dynamic
# error at the x=1 statement (b/33975425). I think we need to simplify
# the resolver algorithm to what we have implemented.
def use_before_def():
print(x) # dynamic error: local var referenced before assignment
printok[0] = True
x = 1 # makes 'x' local
assert.fails(use_before_def, 'local variable x referenced before assignment')
assert.true(not printok[0]) # execution of print statement failed
---
x = [1]
x.extend([2]) # ok
def f():
x += [4] ### "local variable x referenced before assignment"
f()
---
z += 3 ### "global variable z referenced before assignment"
---
load("assert.star", "assert")
# It's ok to define a global that shadows a built-in...
list = []
assert.eq(type(list), "list")
# ...but then all uses refer to the global,
# even if they occur before the binding use.
# See github.com/google/skylark/issues/116.
assert.fails(lambda: tuple, "global variable tuple referenced before assignment")
tuple = ()
---
# option:set
# Same as above, but set is dialect-specific;
# we shouldn't notice any difference.
load("assert.star", "assert")
set = [1, 2, 3]
assert.eq(type(set), "list")
# As in Python 2 and Python 3,
# all 'in x' expressions in a comprehension are evaluated
# in the comprehension's lexical block, except the first,
# which is resolved in the outer block.
x = [[1, 2]]
assert.eq([x for x in x for y in x],
[[1, 2], [1, 2]])
---
# A comprehension establishes a single new lexical block,
# not one per 'for' clause.
x = [1, 2]
_ = [x for _ in [3] for x in x] ### "local variable x referenced before assignment"
---
load("assert.star", "assert")
# assign singleton sequence to 1-tuple
(x,) = (1,)
assert.eq(x, 1)
(y,) = [1]
assert.eq(y, 1)
# assign 1-tuple to variable
z = (1,)
assert.eq(type(z), "tuple")
assert.eq(len(z), 1)
assert.eq(z[0], 1)
# assign value to parenthesized variable
(a) = 1
assert.eq(a, 1)
---
# assignment to/from fields.
load("assert.star", "assert", "freeze")
hf = hasfields()
hf.x = 1
assert.eq(hf.x, 1)
hf.x = [1, 2]
hf.x += [3, 4]
assert.eq(hf.x, [1, 2, 3, 4])
freeze(hf)
def setX(hf):
hf.x = 2
def setY(hf):
hf.y = 3
assert.fails(lambda: setX(hf), "cannot set field on a frozen hasfields")
assert.fails(lambda: setY(hf), "cannot set field on a frozen hasfields")
---
# destucturing assignment in a for loop.
load("assert.star", "assert")
def f():
res = []
for (x, y), z in [(["a", "b"], 3), (["c", "d"], 4)]:
res.append((x, y, z))
return res
assert.eq(f(), [("a", "b", 3), ("c", "d", 4)])
def g():
a = {}
for i, a[i] in [("one", 1), ("two", 2)]:
pass
return a
assert.eq(g(), {"one": 1, "two": 2})
---
# parenthesized LHS in augmented assignment (success)
# option:globalreassign
load("assert.star", "assert")
a = 5
(a) += 3
assert.eq(a, 8)
---
# parenthesized LHS in augmented assignment (error)
(a) += 5 ### "global variable a referenced before assignment"
---
# option:globalreassign
load("assert.star", "assert")
assert = 1
load("assert.star", "assert")
---
# option:globalreassign option:loadbindsglobally
load("assert.star", "assert")
assert = 1
load("assert.star", "assert")
---
# option:loadbindsglobally
_ = assert ### "global variable assert referenced before assignment"
load("assert.star", "assert")
---
_ = assert ### "local variable assert referenced before assignment"
load("assert.star", "assert")
---
def f(): assert.eq(1, 1) # forward ref OK
load("assert.star", "assert")
f()
---
# option:loadbindsglobally
def f(): assert.eq(1, 1) # forward ref OK
load("assert.star", "assert")
f()
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/benchmark.star
Vendored
+167
View File
@@ -0,0 +1,167 @@
# Benchmarks of Starlark execution
# option:set
def bench_range_construction(b):
for _ in range(b.n):
range(200)
def bench_range_iteration(b):
for _ in range(b.n):
for x in range(200):
pass
# Make a 2-level call tree of 100 * 100 calls.
def bench_calling(b):
list = range(100)
def g():
for x in list:
pass
def f():
for x in list:
g()
for _ in range(b.n):
f()
# Measure overhead of calling a trivial built-in method.
emptydict = {}
range1000 = range(1000)
def bench_builtin_method(b):
for _ in range(b.n):
for _ in range1000:
emptydict.get(None)
def bench_int(b):
for _ in range(b.n):
a = 0
for _ in range1000:
a += 1
def bench_bigint(b):
for _ in range(b.n):
a = 1 << 31 # maxint32 + 1
for _ in range1000:
a += 1
def bench_gauss(b):
# Sum of arithmetic series. All results fit in int32.
for _ in range(b.n):
acc = 0
for x in range(92000):
acc += x
def bench_mix(b):
"Benchmark of a simple mix of computation (for, if, arithmetic, comprehension)."
for _ in range(b.n):
x = 0
for i in range(50):
if i:
x += 1
a = [x for x in range(i)]
largedict = {str(v): v for v in range(1000)}
def bench_dict_equal(b):
"Benchmark of dict equality operation."
for _ in range(b.n):
if largedict != largedict:
fail("invalid comparison")
largeset = set([v for v in range(1000)])
def bench_set_equal(b):
"Benchmark of set union operation."
for _ in range(b.n):
if largeset != largeset:
fail("invalid comparison")
flat = { "int": 1, "float": 0.2, "string": "string", "list": [], "bool": True, "nil": None, "tuple": (1, 2, 3) }
deep = {
"type": "int",
"value": 1,
"next": {
"type": "float",
"value": 0.2,
"next": {
"type": "string",
"value": "string",
"next": {
"type": "list",
"value": [ 1, "", True, None, (1, 2) ],
"next": {
"type": "bool",
"value": True,
"next": {
"type": "tuple",
"value": (1, 2.0, "3"),
"next": None
}
}
}
}
}
}
deep_list = [ deep for _ in range(100) ]
def bench_to_json_flat_mixed(b):
"Benchmark json.encode builtin with flat mixed input"
for _ in range(b.n):
json.encode(flat)
def bench_to_json_flat_big(b):
"Benchmark json.encode builtin with big flat integer input"
for _ in range(b.n):
json.encode(largedict)
def bench_to_json_deep(b):
"Benchmark json.encode builtin with deep input"
for _ in range(b.n):
json.encode(deep)
def bench_to_json_deep_list(b):
"Benchmark json.encode builtin with a list of deep input"
for _ in range(b.n):
json.encode(deep)
def bench_issubset_unique_large_small(b):
"Benchmark set.issubset builtin"
s = set(range(10000))
for _ in range(b.n):
s.issubset(range(1000))
def bench_issubset_unique_small_large(b):
"Benchmark set.issubset builtin"
s = set(range(1000))
for _ in range(b.n):
s.issubset(range(10000))
def bench_issubset_unique_same(b):
"Benchmark set.issubset builtin"
s = set(range(1000))
for _ in range(b.n):
s.issubset(range(1000))
def bench_issubset_duplicate_large_small(b):
"Benchmark set.issubset builtin"
s = set(range(10000))
l = list(range(200)) * 5
for _ in range(b.n):
s.issubset(range(1000))
def bench_issubset_duplicate_small_large(b):
"Benchmark set.issubset builtin"
s = set(range(1000))
l = list(range(2000)) * 5
for _ in range(b.n):
s.issubset(l)
def bench_issubset_duplicate_same(b):
"Benchmark set.issubset builtin"
s = set(range(1000))
l = list(range(200)) * 5
for _ in range(b.n):
s.issubset(l)
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/bool.star
Vendored
+62
View File
@@ -0,0 +1,62 @@
# Tests of Starlark 'bool'
load("assert.star", "assert")
# truth
assert.true(True)
assert.true(not False)
assert.true(not not True)
assert.true(not not 1 >= 1)
# precedence of not
assert.true(not not 2 > 1)
# assert.true(not (not 2) > 1) # TODO(adonovan): fix: gives error for False > 1.
# assert.true(not ((not 2) > 1)) # TODO(adonovan): fix
# assert.true(not ((not (not 2)) > 1)) # TODO(adonovan): fix
# assert.true(not not not (2 > 1))
# bool conversion
assert.eq(
[bool(), bool(1), bool(0), bool("hello"), bool("")],
[False, True, False, True, False],
)
# comparison
assert.true(None == None)
assert.true(None != False)
assert.true(None != True)
assert.eq(1 == 1, True)
assert.eq(1 == 2, False)
assert.true(False == False)
assert.true(True == True)
# ordered comparison
assert.true(False < True)
assert.true(False <= True)
assert.true(False <= False)
assert.true(True > False)
assert.true(True >= False)
assert.true(True >= True)
# conditional expression
assert.eq(1 if 3 > 2 else 0, 1)
assert.eq(1 if "foo" else 0, 1)
assert.eq(1 if "" else 0, 0)
# short-circuit evaluation of 'and' and 'or':
# 'or' yields the first true operand, or the last if all are false.
assert.eq(0 or "" or [] or 0, 0)
assert.eq(0 or "" or [] or 123 or 1 // 0, 123)
assert.fails(lambda : 0 or "" or [] or 0 or 1 // 0, "division by zero")
# 'and' yields the first false operand, or the last if all are true.
assert.eq(1 and "a" and [1] and 123, 123)
assert.eq(1 and "a" and [1] and 0 and 1 // 0, 0)
assert.fails(lambda : 1 and "a" and [1] and 123 and 1 // 0, "division by zero")
# Built-ins that want a bool want an actual bool, not a truth value.
# See github.com/bazelbuild/starlark/issues/30
assert.eq(''.splitlines(True), [])
assert.fails(lambda: ''.splitlines(1), 'got int, want bool')
assert.fails(lambda: ''.splitlines("hello"), 'got string, want bool')
assert.fails(lambda: ''.splitlines(0.0), 'got float, want bool')
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/builtins.star
Vendored
+240
View File
@@ -0,0 +1,240 @@
# Tests of Starlark built-in functions
# option:set
load("assert.star", "assert")
# len
assert.eq(len([1, 2, 3]), 3)
assert.eq(len((1, 2, 3)), 3)
assert.eq(len({1: 2}), 1)
assert.fails(lambda: len(1), "int.*has no len")
# and, or
assert.eq(123 or "foo", 123)
assert.eq(0 or "foo", "foo")
assert.eq(123 and "foo", "foo")
assert.eq(0 and "foo", 0)
none = None
_1 = none and none[0] # rhs is not evaluated
_2 = (not none) or none[0] # rhs is not evaluated
# abs
assert.eq(abs(2.0), 2.0)
assert.eq(abs(0.0), 0.0)
assert.eq(abs(-2.0), 2.0)
assert.eq(abs(2), 2)
assert.eq(abs(0), 0)
assert.eq(abs(-2), 2)
assert.eq(abs(float("inf")), float("inf"))
assert.eq(abs(float("-inf")), float("inf"))
assert.eq(abs(float("nan")), float("nan"))
assert.fails(lambda: abs("0"), "got string, want int or float")
maxint32 = (1 << 31) - 1
assert.eq(abs(+123 * maxint32), +123 * maxint32)
assert.eq(abs(-123 * maxint32), +123 * maxint32)
# any, all
assert.true(all([]))
assert.true(all([1, True, "foo"]))
assert.true(not all([1, True, ""]))
assert.true(not any([]))
assert.true(any([0, False, "foo"]))
assert.true(not any([0, False, ""]))
# in
assert.true(3 in [1, 2, 3])
assert.true(4 not in [1, 2, 3])
assert.true(3 in (1, 2, 3))
assert.true(4 not in (1, 2, 3))
assert.fails(lambda: 3 in "foo", "in.*requires string as left operand")
assert.true(123 in {123: ""})
assert.true(456 not in {123:""})
assert.true([] not in {123: ""})
# sorted
assert.eq(sorted([42, 123, 3]), [3, 42, 123])
assert.eq(sorted([42, 123, 3], reverse=True), [123, 42, 3])
assert.eq(sorted(["wiz", "foo", "bar"]), ["bar", "foo", "wiz"])
assert.eq(sorted(["wiz", "foo", "bar"], reverse=True), ["wiz", "foo", "bar"])
assert.fails(lambda: sorted([1, 2, None, 3]), "int < NoneType not implemented")
assert.fails(lambda: sorted([1, "one"]), "string < int not implemented")
# custom key function
assert.eq(sorted(["two", "three", "four"], key=len),
["two", "four", "three"])
assert.eq(sorted(["two", "three", "four"], key=len, reverse=True),
["three", "four", "two"])
assert.fails(lambda: sorted([1, 2, 3], key=None), "got NoneType, want callable")
# sort is stable
pairs = [(4, 0), (3, 1), (4, 2), (2, 3), (3, 4), (1, 5), (2, 6), (3, 7)]
assert.eq(sorted(pairs, key=lambda x: x[0]),
[(1, 5),
(2, 3), (2, 6),
(3, 1), (3, 4), (3, 7),
(4, 0), (4, 2)])
assert.fails(lambda: sorted(1), 'sorted: for parameter iterable: got int, want iterable')
# reversed
assert.eq(reversed([1, 144, 81, 16]), [16, 81, 144, 1])
# set
assert.contains(set([1, 2, 3]), 1)
assert.true(4 not in set([1, 2, 3]))
assert.eq(len(set([1, 2, 3])), 3)
assert.eq(sorted([x for x in set([1, 2, 3])]), [1, 2, 3])
# dict
assert.eq(dict([(1, 2), (3, 4)]), {1: 2, 3: 4})
assert.eq(dict([(1, 2), (3, 4)], foo="bar"), {1: 2, 3: 4, "foo": "bar"})
assert.eq(dict({1:2, 3:4}), {1: 2, 3: 4})
assert.eq(dict({1:2, 3:4}.items()), {1: 2, 3: 4})
# range
assert.eq("range", type(range(10)))
assert.eq("range(10)", str(range(0, 10, 1)))
assert.eq("range(1, 10)", str(range(1, 10)))
assert.eq(range(0, 5, 10), range(0, 5, 11))
assert.eq("range(0, 10, -1)", str(range(0, 10, -1)))
assert.fails(lambda: {range(10): 10}, "unhashable: range")
assert.true(bool(range(1, 2)))
assert.true(not(range(2, 1))) # an empty range is false
assert.eq([x*x for x in range(5)], [0, 1, 4, 9, 16])
assert.eq(list(range(5)), [0, 1, 2, 3, 4])
assert.eq(list(range(-5)), [])
assert.eq(list(range(2, 5)), [2, 3, 4])
assert.eq(list(range(5, 2)), [])
assert.eq(list(range(-2, -5)), [])
assert.eq(list(range(-5, -2)), [-5, -4, -3])
assert.eq(list(range(2, 10, 3)), [2, 5, 8])
assert.eq(list(range(10, 2, -3)), [10, 7, 4])
assert.eq(list(range(-2, -10, -3)), [-2, -5, -8])
assert.eq(list(range(-10, -2, 3)), [-10, -7, -4])
assert.eq(list(range(10, 2, -1)), [10, 9, 8, 7, 6, 5, 4, 3])
assert.eq(list(range(5)[1:]), [1, 2, 3, 4])
assert.eq(len(range(5)[1:]), 4)
assert.eq(list(range(5)[:2]), [0, 1])
assert.eq(list(range(10)[1:]), [1, 2, 3, 4, 5, 6, 7, 8, 9])
assert.eq(list(range(10)[1:9:2]), [1, 3, 5, 7])
assert.eq(list(range(10)[1:10:2]), [1, 3, 5, 7, 9])
assert.eq(list(range(10)[1:11:2]), [1, 3, 5, 7, 9])
assert.eq(list(range(10)[::-2]), [9, 7, 5, 3, 1])
assert.eq(list(range(0, 10, 2)[::2]), [0, 4, 8])
assert.eq(list(range(0, 10, 2)[::-2]), [8, 4, 0])
# range() is limited by the width of the Go int type (int32 or int64).
assert.fails(lambda: range(1<<64), "... out of range .want value in signed ..-bit range")
assert.eq(len(range(0x7fffffff)), 0x7fffffff) # O(1)
# Two ranges compare equal if they denote the same sequence:
assert.eq(range(0), range(2, 1, 3)) # []
assert.eq(range(0, 3, 2), range(0, 4, 2)) # [0, 2]
assert.ne(range(1, 10), range(2, 10))
assert.fails(lambda: range(0) < range(0), "range < range not implemented")
# <number> in <range>
assert.contains(range(3), 1)
assert.contains(range(3), 2.0) # acts like 2
assert.fails(lambda: True in range(3), "requires integer.*not bool") # bools aren't numbers
assert.fails(lambda: "one" in range(10), "requires integer.*not string")
assert.true(4 not in range(4))
assert.true(1e15 not in range(4)) # too big for int32
assert.true(1e100 not in range(4)) # too big for int64
# https://github.com/google/starlark-go/issues/116
assert.fails(lambda: range(0, 0, 2)[:][0], "index 0 out of range: empty range")
# list
assert.eq(list("abc".elems()), ["a", "b", "c"])
assert.eq(sorted(list({"a": 1, "b": 2})), ['a', 'b'])
# min, max
assert.eq(min(5, -2, 1, 7, 3), -2)
assert.eq(max(5, -2, 1, 7, 3), 7)
assert.eq(min([5, -2, 1, 7, 3]), -2)
assert.eq(min("one", "two", "three", "four"), "four")
assert.eq(max("one", "two", "three", "four"), "two")
assert.fails(min, "min requires at least one positional argument")
assert.fails(lambda: min(1), "not iterable")
assert.fails(lambda: min([]), "empty")
assert.eq(min(5, -2, 1, 7, 3, key=lambda x: x*x), 1) # min absolute value
assert.eq(min(5, -2, 1, 7, 3, key=lambda x: -x), 7) # min negated value
# enumerate
assert.eq(enumerate("abc".elems()), [(0, "a"), (1, "b"), (2, "c")])
assert.eq(enumerate([False, True, None], 42), [(42, False), (43, True), (44, None)])
# zip
assert.eq(zip(), [])
assert.eq(zip([]), [])
assert.eq(zip([1, 2, 3]), [(1,), (2,), (3,)])
assert.eq(zip("".elems()), [])
assert.eq(zip("abc".elems(),
list("def".elems()),
"hijk".elems()),
[("a", "d", "h"), ("b", "e", "i"), ("c", "f", "j")])
z1 = [1]
assert.eq(zip(z1), [(1,)])
z1.append(2)
assert.eq(zip(z1), [(1,), (2,)])
assert.fails(lambda: zip(z1, 1), "zip: argument #2 is not iterable: int")
z1.append(3)
# dir for builtin_function_or_method
assert.eq(dir(None), [])
assert.eq(dir({})[:3], ["clear", "get", "items"]) # etc
assert.eq(dir(1), [])
assert.eq(dir([])[:3], ["append", "clear", "extend"]) # etc
# hasattr, getattr, dir
# hasfields is an application-defined type defined in eval_test.go.
hf = hasfields()
assert.eq(dir(hf), [])
assert.true(not hasattr(hf, "x"))
assert.fails(lambda: getattr(hf, "x"), "no .x field or method")
assert.eq(getattr(hf, "x", 42), 42)
hf.x = 1
assert.true(hasattr(hf, "x"))
assert.eq(getattr(hf, "x"), 1)
assert.eq(hf.x, 1)
hf.x = 2
assert.eq(getattr(hf, "x"), 2)
assert.eq(hf.x, 2)
# built-in types can have attributes (methods) too.
myset = set([])
assert.eq(dir(myset), ["add", "clear", "difference", "discard", "intersection", "issubset", "issuperset", "pop", "remove", "symmetric_difference", "union"])
assert.true(hasattr(myset, "union"))
assert.true(not hasattr(myset, "onion"))
assert.eq(str(getattr(myset, "union")), "<built-in method union of set value>")
assert.fails(lambda: getattr(myset, "onion"), "no .onion field or method")
assert.eq(getattr(myset, "onion", 42), 42)
# dir returns a new, sorted, mutable list
assert.eq(sorted(dir("")), dir("")) # sorted
dir("").append("!") # mutable
assert.true("!" not in dir("")) # new
# error messages should suggest spelling corrections
hf.one = 1
hf.two = 2
hf.three = 3
hf.forty_five = 45
assert.fails(lambda: hf.One, 'no .One field.*did you mean .one')
assert.fails(lambda: hf.oone, 'no .oone field.*did you mean .one')
assert.fails(lambda: hf.FortyFive, 'no .FortyFive field.*did you mean .forty_five')
assert.fails(lambda: hf.trhee, 'no .trhee field.*did you mean .three')
assert.fails(lambda: hf.thirty, 'no .thirty field or method$') # no suggestion
# spell check in setfield too
def setfield(): hf.noForty_Five = 46 # "no" prefix => SetField returns NoSuchField
assert.fails(setfield, 'no .noForty_Five field.*did you mean .forty_five')
# repr
assert.eq(repr(1), "1")
assert.eq(repr("x"), '"x"')
assert.eq(repr(["x", 1]), '["x", 1]')
# fail
---
fail() ### `fail: $`
x = 1//0 # unreachable
---
fail(1) ### `fail: 1`
---
fail(1, 2, 3) ### `fail: 1 2 3`
---
fail(1, 2, 3, sep="/") ### `fail: 1/2/3`
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/bytes.star
Vendored
+159
View File
@@ -0,0 +1,159 @@
# Tests of 'bytes' (immutable byte strings).
load("assert.star", "assert")
# bytes(string) -- UTF-k to UTF-8 transcoding with U+FFFD replacement
hello = bytes("hello, 世界")
goodbye = bytes("goodbye")
empty = bytes("")
nonprinting = bytes("\t\n\x7F\u200D") # TAB, NEWLINE, DEL, ZERO_WIDTH_JOINER
assert.eq(bytes("hello, 世界"[:-1]), b"hello, 世��")
# bytes(iterable of int) -- construct from numeric byte values
assert.eq(bytes([65, 66, 67]), b"ABC")
assert.eq(bytes((65, 66, 67)), b"ABC")
assert.eq(bytes([0xf0, 0x9f, 0x98, 0xbf]), b"😿")
assert.fails(lambda: bytes([300]),
"at index 0, 300 out of range .want value in unsigned 8-bit range")
assert.fails(lambda: bytes([b"a"]),
"at index 0, got bytes, want int")
assert.fails(lambda: bytes(1), "want string, bytes, or iterable of ints")
# literals
assert.eq(b"hello, 世界", hello)
assert.eq(b"goodbye", goodbye)
assert.eq(b"", empty)
assert.eq(b"\t\n\x7F\u200D", nonprinting)
assert.ne("abc", b"abc")
assert.eq(b"\012\xff\u0400\U0001F63F", b"\n\xffЀ😿") # see scanner tests for more
assert.eq(rb"\r\n\t", b"\\r\\n\\t") # raw
# type
assert.eq(type(hello), "bytes")
# len
assert.eq(len(hello), 13)
assert.eq(len(goodbye), 7)
assert.eq(len(empty), 0)
assert.eq(len(b"A"), 1)
assert.eq(len(b"Ѐ"), 2)
assert.eq(len(b"世"), 3)
assert.eq(len(b"😿"), 4)
# truth
assert.true(hello)
assert.true(goodbye)
assert.true(not empty)
# str(bytes) does UTF-8 to UTF-k transcoding.
# TODO(adonovan): specify.
assert.eq(str(hello), "hello, 世界")
assert.eq(str(hello[:-1]), "hello, 世��") # incomplete UTF-8 encoding => U+FFFD
assert.eq(str(goodbye), "goodbye")
assert.eq(str(empty), "")
assert.eq(str(nonprinting), "\t\n\x7f\u200d")
assert.eq(str(b"\xED\xB0\x80"), "���") # UTF-8 encoding of unpaired surrogate => U+FFFD x 3
# repr
assert.eq(repr(hello), r'b"hello, 世界"')
assert.eq(repr(hello[:-1]), r'b"hello, 世\xe7\x95"') # (incomplete UTF-8 encoding )
assert.eq(repr(goodbye), 'b"goodbye"')
assert.eq(repr(empty), 'b""')
assert.eq(repr(nonprinting), 'b"\\t\\n\\x7f\\u200d"')
# equality
assert.eq(hello, hello)
assert.ne(hello, goodbye)
assert.eq(b"goodbye", goodbye)
# ordered comparison
assert.lt(b"abc", b"abd")
assert.lt(b"abc", b"abcd")
assert.lt(b"\x7f", b"\x80") # bytes compare as uint8, not int8
# bytes are dict-hashable
dict = {hello: 1, goodbye: 2}
dict[b"goodbye"] = 3
assert.eq(len(dict), 2)
assert.eq(dict[goodbye], 3)
# hash(bytes) is 32-bit FNV-1a.
assert.eq(hash(b""), 0x811c9dc5)
assert.eq(hash(b"a"), 0xe40c292c)
assert.eq(hash(b"ab"), 0x4d2505ca)
assert.eq(hash(b"abc"), 0x1a47e90b)
# indexing
assert.eq(goodbye[0], b"g")
assert.eq(goodbye[-1], b"e")
assert.fails(lambda: goodbye[100], "out of range")
# slicing
assert.eq(goodbye[:4], b"good")
assert.eq(goodbye[4:], b"bye")
assert.eq(goodbye[::2], b"gobe")
assert.eq(goodbye[3:4], b"d") # special case: len=1
assert.eq(goodbye[4:4], b"") # special case: len=0
# bytes in bytes
assert.eq(b"bc" in b"abcd", True)
assert.eq(b"bc" in b"dcab", False)
assert.fails(lambda: "bc" in b"dcab", "requires bytes or int as left operand, not string")
# int in bytes
assert.eq(97 in b"abc", True) # 97='a'
assert.eq(100 in b"abc", False) # 100='d'
assert.fails(lambda: 256 in b"abc", "int in bytes: 256 out of range")
assert.fails(lambda: -1 in b"abc", "int in bytes: -1 out of range")
# ord TODO(adonovan): specify
assert.eq(ord(b"a"), 97)
assert.fails(lambda: ord(b"ab"), "ord: bytes has length 2, want 1")
assert.fails(lambda: ord(b""), "ord: bytes has length 0, want 1")
# repeat (bytes * int)
assert.eq(goodbye * 3, b"goodbyegoodbyegoodbye")
assert.eq(3 * goodbye, b"goodbyegoodbyegoodbye")
# elems() returns an iterable value over 1-byte substrings.
assert.eq(type(hello.elems()), "bytes.elems")
assert.eq(str(hello.elems()), "b\"hello, 世界\".elems()")
assert.eq(list(hello.elems()), [104, 101, 108, 108, 111, 44, 32, 228, 184, 150, 231, 149, 140])
assert.eq(bytes([104, 101, 108, 108, 111, 44, 32, 228, 184, 150, 231, 149, 140]), hello)
assert.eq(list(goodbye.elems()), [103, 111, 111, 100, 98, 121, 101])
assert.eq(list(empty.elems()), [])
assert.eq(bytes(hello.elems()), hello) # bytes(iterable) is dual to bytes.elems()
# x[i] = ...
def f():
b"abc"[1] = b"B"
assert.fails(f, "bytes.*does not support.*assignment")
# TODO(adonovan): the specification is not finalized in many areas:
# - chr, ord functions
# - encoding/decoding bytes to string.
# - methods: find, index, split, etc.
#
# Summary of string operations (put this in spec).
#
# string to number:
# - bytes[i] returns numeric value of ith byte.
# - ord(string) returns numeric value of sole code point in string.
# - ord(string[i]) is not a useful operation: fails on non-ASCII; see below.
# Q. Perhaps ord should return the first (not sole) code point? Then it becomes a UTF-8 decoder.
# Perhaps ord(string, index=int) should apply the index and relax the len=1 check.
# - string.codepoint() iterates over 1-codepoint substrings.
# - string.codepoint_ords() iterates over numeric values of code points in string.
# - string.elems() iterates over 1-element (UTF-k code) substrings.
# - string.elem_ords() iterates over numeric UTF-k code values.
# - string.elem_ords()[i] returns numeric value of ith element (UTF-k code).
# - string.elems()[i] returns substring of a single element (UTF-k code).
# - int(string) parses string as decimal (or other) numeric literal.
#
# number to string:
# - chr(int) returns string, UTF-k encoding of Unicode code point (like Python).
# Redundant with '%c' % int (which Python2 calls 'unichr'.)
# - bytes(chr(int)) returns byte string containing UTF-8 encoding of one code point.
# - bytes([int]) returns 1-byte string (with regrettable list allocation).
# - str(int) - format number as decimal.
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/control.star
Vendored
+64
View File
@@ -0,0 +1,64 @@
# Tests of Starlark control flow
load("assert.star", "assert")
def controlflow():
# elif
x = 0
if True:
x=1
elif False:
assert.fail("else of true")
else:
assert.fail("else of else of true")
assert.true(x)
x = 0
if False:
assert.fail("then of false")
elif True:
x = 1
else:
assert.fail("else of true")
assert.true(x)
x = 0
if False:
assert.fail("then of false")
elif False:
assert.fail("then of false")
else:
x = 1
assert.true(x)
controlflow()
def loops():
y = ""
for x in [1, 2, 3, 4, 5]:
if x == 2:
continue
if x == 4:
break
y = y + str(x)
return y
assert.eq(loops(), "13")
# return
g = 123
def f(x):
for g in (1, 2, 3):
if g == x:
return g
assert.eq(f(2), 2)
assert.eq(f(4), None) # falling off end => return None
assert.eq(g, 123) # unchanged by local use of g in function
# infinite sequences
def fib(n):
seq = []
for x in fibonacci: # fibonacci is an infinite iterable defined in eval_test.go
if len(seq) == n:
break
seq.append(x)
return seq
assert.eq(fib(10), [0, 1, 1, 2, 3, 5, 8, 13, 21, 34])
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/dict.star
Vendored
+321
View File
@@ -0,0 +1,321 @@
# Tests of Starlark 'dict'
load("assert.star", "assert", "freeze")
# literals
assert.eq({}, {})
assert.eq({"a": 1}, {"a": 1})
assert.eq({"a": 1,}, {"a": 1})
# truth
assert.true({False: False})
assert.true(not {})
# dict + dict is no longer supported.
assert.fails(lambda: {"a": 1} + {"b": 2}, 'unknown binary op: dict \\+ dict')
# dict comprehension
assert.eq({x: x*x for x in range(3)}, {0: 0, 1: 1, 2: 4})
# dict.pop
x6 = {"a": 1, "b": 2}
assert.eq(x6.pop("a"), 1)
assert.eq(str(x6), '{"b": 2}')
assert.fails(lambda: x6.pop("c"), "pop: missing key")
assert.eq(x6.pop("c", 3), 3)
assert.eq(x6.pop("c", None), None) # default=None tests an edge case of UnpackArgs
assert.eq(x6.pop("b"), 2)
assert.eq(len(x6), 0)
# dict.popitem
x7 = {"a": 1, "b": 2}
assert.eq([x7.popitem(), x7.popitem()], [("a", 1), ("b", 2)])
assert.fails(x7.popitem, "empty dict")
assert.eq(len(x7), 0)
# dict.keys, dict.values
x8 = {"a": 1, "b": 2}
assert.eq(x8.keys(), ["a", "b"])
assert.eq(x8.values(), [1, 2])
# equality
assert.eq({"a": 1, "b": 2}, {"a": 1, "b": 2})
assert.eq({"a": 1, "b": 2,}, {"a": 1, "b": 2})
assert.eq({"a": 1, "b": 2}, {"b": 2, "a": 1})
# insertion order is preserved
assert.eq(dict([("a", 0), ("b", 1), ("c", 2), ("b", 3)]).keys(), ["a", "b", "c"])
assert.eq(dict([("b", 0), ("a", 1), ("b", 2), ("c", 3)]).keys(), ["b", "a", "c"])
assert.eq(dict([("b", 0), ("a", 1), ("b", 2), ("c", 3)])["b"], 2)
# ...even after rehashing (which currently occurs after key 'i'):
small = dict([("a", 0), ("b", 1), ("c", 2)])
small.update([("d", 4), ("e", 5), ("f", 6), ("g", 7), ("h", 8), ("i", 9), ("j", 10), ("k", 11)])
assert.eq(small.keys(), ["a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k"])
# Duplicate keys are not permitted in dictionary expressions (see b/35698444).
# (Nor in keyword args to function calls---checked by resolver.)
assert.fails(lambda: {"aa": 1, "bb": 2, "cc": 3, "bb": 4}, 'duplicate key: "bb"')
# Check that even with many positional args, keyword collisions are detected.
assert.fails(lambda: dict({'b': 3}, a=4, **dict(a=5)), 'dict: duplicate keyword arg: "a"')
assert.fails(lambda: dict({'a': 2, 'b': 3}, a=4, **dict(a=5)), 'dict: duplicate keyword arg: "a"')
# positional/keyword arg key collisions are ok
assert.eq(dict((['a', 2], ), a=4), {'a': 4})
assert.eq(dict((['a', 2], ['a', 3]), a=4), {'a': 4})
# index
def setIndex(d, k, v):
d[k] = v
x9 = {}
assert.fails(lambda: x9["a"], 'key "a" not in dict')
x9["a"] = 1
assert.eq(x9["a"], 1)
assert.eq(x9, {"a": 1})
assert.fails(lambda: setIndex(x9, [], 2), 'unhashable type: list')
freeze(x9)
assert.fails(lambda: setIndex(x9, "a", 3), 'cannot insert into frozen hash table')
x9a = {}
x9a[1, 2] = 3 # unparenthesized tuple is allowed here
assert.eq(x9a.keys()[0], (1, 2))
# dict.get
x10 = {"a": 1}
assert.eq(x10.get("a"), 1)
assert.eq(x10.get("b"), None)
assert.eq(x10.get("a", 2), 1)
assert.eq(x10.get("b", 2), 2)
# dict.clear
x11 = {"a": 1}
assert.contains(x11, "a")
assert.eq(x11["a"], 1)
x11.clear()
assert.fails(lambda: x11["a"], 'key "a" not in dict')
assert.true("a" not in x11)
freeze(x11)
assert.fails(x11.clear, "cannot clear frozen hash table")
# dict.setdefault
x12 = {"a": 1}
assert.eq(x12.setdefault("a"), 1)
assert.eq(x12["a"], 1)
assert.eq(x12.setdefault("b"), None)
assert.eq(x12["b"], None)
assert.eq(x12.setdefault("c", 2), 2)
assert.eq(x12["c"], 2)
assert.eq(x12.setdefault("c", 3), 2)
assert.eq(x12["c"], 2)
freeze(x12)
assert.eq(x12.setdefault("a", 1), 1) # no change, no error
assert.fails(lambda: x12.setdefault("d", 1), "cannot insert into frozen hash table")
# dict.update
x13 = {"a": 1}
x13.update(a=2, b=3)
assert.eq(x13, {"a": 2, "b": 3})
x13.update([("b", 4), ("c", 5)])
assert.eq(x13, {"a": 2, "b": 4, "c": 5})
x13.update({"c": 6, "d": 7})
assert.eq(x13, {"a": 2, "b": 4, "c": 6, "d": 7})
freeze(x13)
assert.fails(lambda: x13.update({"a": 8}), "cannot insert into frozen hash table")
# dict as a sequence
#
# for loop
x14 = {1:2, 3:4}
def keys(dict):
keys = []
for k in dict: keys.append(k)
return keys
assert.eq(keys(x14), [1, 3])
#
# comprehension
assert.eq([x for x in x14], [1, 3])
#
# varargs
def varargs(*args): return args
x15 = {"one": 1}
assert.eq(varargs(*x15), ("one",))
# kwargs parameter does not alias the **kwargs dict
def kwargs(**kwargs): return kwargs
x16 = kwargs(**x15)
assert.eq(x16, x15)
x15["two"] = 2 # mutate
assert.ne(x16, x15)
# iterator invalidation
def iterator1():
dict = {1:1, 2:1}
for k in dict:
dict[2*k] = dict[k]
assert.fails(iterator1, "insert.*during iteration")
def iterator2():
dict = {1:1, 2:1}
for k in dict:
dict.pop(k)
assert.fails(iterator2, "delete.*during iteration")
def iterator3():
def f(d):
d[3] = 3
dict = {1:1, 2:1}
_ = [f(dict) for x in dict]
assert.fails(iterator3, "insert.*during iteration")
# This assignment is not a modification-during-iteration:
# the sequence x should be completely iterated before
# the assignment occurs.
def f():
x = {1:2, 2:4}
a, x[0] = x
assert.eq(a, 1)
assert.eq(x, {1: 2, 2: 4, 0: 2})
f()
# Regression test for a bug in hashtable.delete
def test_delete():
d = {}
# delete tail first
d["one"] = 1
d["two"] = 2
assert.eq(str(d), '{"one": 1, "two": 2}')
d.pop("two")
assert.eq(str(d), '{"one": 1}')
d.pop("one")
assert.eq(str(d), '{}')
# delete head first
d["one"] = 1
d["two"] = 2
assert.eq(str(d), '{"one": 1, "two": 2}')
d.pop("one")
assert.eq(str(d), '{"two": 2}')
d.pop("two")
assert.eq(str(d), '{}')
# delete middle
d["one"] = 1
d["two"] = 2
d["three"] = 3
assert.eq(str(d), '{"one": 1, "two": 2, "three": 3}')
d.pop("two")
assert.eq(str(d), '{"one": 1, "three": 3}')
d.pop("three")
assert.eq(str(d), '{"one": 1}')
d.pop("one")
assert.eq(str(d), '{}')
test_delete()
# Regression test for github.com/google/starlark-go/issues/128.
assert.fails(lambda: dict(None), 'got NoneType, want iterable')
assert.fails(lambda: {}.update(None), 'got NoneType, want iterable')
---
# Verify position of an "unhashable key" error in a dict literal.
_ = {
"one": 1,
["two"]: 2, ### "unhashable type: list"
"three": 3,
}
---
# Verify position of a "duplicate key" error in a dict literal.
_ = {
"one": 1,
"one": 1, ### `duplicate key: "one"`
"three": 3,
}
---
# Verify position of an "unhashable key" error in a dict comprehension.
_ = {
k: v ### "unhashable type: list"
for k, v in [
("one", 1),
(["two"], 2),
("three", 3),
]
}
---
# dict | dict (union)
load("assert.star", "assert", "freeze")
empty_dict = dict()
dict_with_a_b = dict(a=1, b=[1, 2])
dict_with_b = dict(b=[1, 2])
dict_with_other_b = dict(b=[3, 4])
assert.eq(empty_dict | dict_with_a_b, dict_with_a_b)
# Verify iteration order.
assert.eq((empty_dict | dict_with_a_b).items(), dict_with_a_b.items())
assert.eq(dict_with_a_b | empty_dict, dict_with_a_b)
assert.eq((dict_with_a_b | empty_dict).items(), dict_with_a_b.items())
assert.eq(dict_with_b | dict_with_a_b, dict_with_a_b)
assert.eq((dict_with_b | dict_with_a_b).items(), dict(b=[1, 2], a=1).items())
assert.eq(dict_with_a_b | dict_with_b, dict_with_a_b)
assert.eq((dict_with_a_b | dict_with_b).items(), dict_with_a_b.items())
assert.eq(dict_with_b | dict_with_other_b, dict_with_other_b)
assert.eq((dict_with_b | dict_with_other_b).items(), dict_with_other_b.items())
assert.eq(dict_with_other_b | dict_with_b, dict_with_b)
assert.eq((dict_with_other_b | dict_with_b).items(), dict_with_b.items())
assert.eq(empty_dict, dict())
assert.eq(dict_with_b, dict(b=[1,2]))
assert.fails(lambda: dict() | [], "unknown binary op: dict [|] list")
# dict |= dict (in-place union)
def test_dict_union_assignment():
x = dict()
saved = x
x |= {"a": 1}
x |= {"b": 2}
x |= {"c": "3", 7: 4}
x |= {"b": "5", "e": 6}
want = {"a": 1, "b": "5", "c": "3", 7: 4, "e": 6}
assert.eq(x, want)
assert.eq(x.items(), want.items())
assert.eq(saved, x) # they are aliases
a = {8: 1, "b": 2}
b = {"b": 1, "c": 6}
c = {"d": 7}
d = {(5, "a"): ("c", 8)}
orig_a, orig_c = a, c
a |= b
c |= a
c |= d
expected_2 = {"d": 7, 8: 1, "b": 1, "c": 6, (5, "a"): ("c", 8)}
assert.eq(c, expected_2)
assert.eq(c.items(), expected_2.items())
assert.eq(b, {"b": 1, "c": 6})
# aliasing:
assert.eq(a, orig_a)
assert.eq(c, orig_c)
a.clear()
c.clear()
assert.eq(a, orig_a)
assert.eq(c, orig_c)
test_dict_union_assignment()
def dict_union_assignment_type_mismatch():
some_dict = dict()
some_dict |= []
assert.fails(dict_union_assignment_type_mismatch, "unknown binary op: dict [|] list")
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/float.star
Vendored
+508
View File
@@ -0,0 +1,508 @@
# Tests of Starlark 'float'
# option:set
load("assert.star", "assert")
# TODO(adonovan): more tests:
# - precision
# - limits
# type
assert.eq(type(0.0), "float")
# truth
assert.true(123.0)
assert.true(-1.0)
assert.true(not 0.0)
assert.true(-1.0e-45)
assert.true(float("NaN"))
# not iterable
assert.fails(lambda: len(0.0), 'has no len')
assert.fails(lambda: [x for x in 0.0], 'float value is not iterable')
# literals
assert.eq(type(1.234), "float")
assert.eq(type(1e10), "float")
assert.eq(type(1e+10), "float")
assert.eq(type(1e-10), "float")
assert.eq(type(1.234e10), "float")
assert.eq(type(1.234e+10), "float")
assert.eq(type(1.234e-10), "float")
# int/float equality
assert.eq(0.0, 0)
assert.eq(0, 0.0)
assert.eq(1.0, 1)
assert.eq(1, 1.0)
assert.true(1.23e45 != 1229999999999999973814869011019624571608236031)
assert.true(1.23e45 == 1229999999999999973814869011019624571608236032)
assert.true(1.23e45 != 1229999999999999973814869011019624571608236033)
assert.true(1229999999999999973814869011019624571608236031 != 1.23e45)
assert.true(1229999999999999973814869011019624571608236032 == 1.23e45)
assert.true(1229999999999999973814869011019624571608236033 != 1.23e45)
# loss of precision
p53 = 1<<53
assert.eq(float(p53-1), p53-1)
assert.eq(float(p53+0), p53+0)
assert.eq(float(p53+1), p53+0) #
assert.eq(float(p53+2), p53+2)
assert.eq(float(p53+3), p53+4) #
assert.eq(float(p53+4), p53+4)
assert.eq(float(p53+5), p53+4) #
assert.eq(float(p53+6), p53+6)
assert.eq(float(p53+7), p53+8) #
assert.eq(float(p53+8), p53+8)
# Regression test for https://github.com/google/starlark-go/issues/375.
maxint64 = (1<<63)-1
assert.eq(int(float(maxint64)), 9223372036854775808)
assert.true(float(p53+1) != p53+1) # comparisons are exact
assert.eq(float(p53+1) - (p53+1), 0) # arithmetic entails rounding
assert.fails(lambda: {123.0: "f", 123: "i"}, "duplicate key: 123")
# equal int/float values have same hash
d = {123.0: "x"}
d[123] = "y"
assert.eq(len(d), 1)
assert.eq(d[123.0], "y")
# literals (mostly covered by scanner tests)
assert.eq(str(0.), "0.0")
assert.eq(str(.0), "0.0")
assert.true(5.0 != 4.999999999999999)
assert.eq(5.0, 4.9999999999999999) # both literals denote 5.0
assert.eq(1.23e45, 1.23 * 1000000000000000000000000000000000000000000000)
assert.eq(str(1.23e-45 - (1.23 / 1000000000000000000000000000000000000000000000)), "-1.5557538194652854e-61")
nan = float("NaN")
inf = float("+Inf")
neginf = float("-Inf")
negzero = (-1e-323 / 10)
# -- arithmetic --
# +float, -float
assert.eq(+(123.0), 123.0)
assert.eq(-(123.0), -123.0)
assert.eq(-(-(123.0)), 123.0)
assert.eq(+(inf), inf)
assert.eq(-(inf), neginf)
assert.eq(-(neginf), inf)
assert.eq(str(-(nan)), "nan")
# +
assert.eq(1.2e3 + 5.6e7, 5.60012e+07)
assert.eq(1.2e3 + 1, 1201)
assert.eq(1 + 1.2e3, 1201)
assert.eq(str(1.2e3 + nan), "nan")
assert.eq(inf + 0, inf)
assert.eq(inf + 1, inf)
assert.eq(inf + inf, inf)
assert.eq(str(inf + neginf), "nan")
# -
assert.eq(1.2e3 - 5.6e7, -5.59988e+07)
assert.eq(1.2e3 - 1, 1199)
assert.eq(1 - 1.2e3, -1199)
assert.eq(str(1.2e3 - nan), "nan")
assert.eq(inf - 0, inf)
assert.eq(inf - 1, inf)
assert.eq(str(inf - inf), "nan")
assert.eq(inf - neginf, inf)
# *
assert.eq(1.5e6 * 2.2e3, 3.3e9)
assert.eq(1.5e6 * 123, 1.845e+08)
assert.eq(123 * 1.5e6, 1.845e+08)
assert.eq(str(1.2e3 * nan), "nan")
assert.eq(str(inf * 0), "nan")
assert.eq(inf * 1, inf)
assert.eq(inf * inf, inf)
assert.eq(inf * neginf, neginf)
# %
assert.eq(100.0 % 7.0, 2)
assert.eq(100.0 % -7.0, -5) # NB: different from Go / Java
assert.eq(-100.0 % 7.0, 5) # NB: different from Go / Java
assert.eq(-100.0 % -7.0, -2)
assert.eq(-100.0 % 7, 5)
assert.eq(100 % 7.0, 2)
assert.eq(str(1.2e3 % nan), "nan")
assert.eq(str(inf % 1), "nan")
assert.eq(str(inf % inf), "nan")
assert.eq(str(inf % neginf), "nan")
# /
assert.eq(str(100.0 / 7.0), "14.285714285714286")
assert.eq(str(100 / 7.0), "14.285714285714286")
assert.eq(str(100.0 / 7), "14.285714285714286")
assert.eq(str(100.0 / nan), "nan")
# //
assert.eq(100.0 // 7.0, 14)
assert.eq(100 // 7.0, 14)
assert.eq(100.0 // 7, 14)
assert.eq(100.0 // -7.0, -15)
assert.eq(100 // -7.0, -15)
assert.eq(100.0 // -7, -15)
assert.eq(str(1 // neginf), "-0.0")
assert.eq(str(100.0 // nan), "nan")
# addition
assert.eq(0.0 + 1.0, 1.0)
assert.eq(1.0 + 1.0, 2.0)
assert.eq(1.25 + 2.75, 4.0)
assert.eq(5.0 + 7.0, 12.0)
assert.eq(5.1 + 7, 12.1) # float + int
assert.eq(7 + 5.1, 12.1) # int + float
# subtraction
assert.eq(5.0 - 7.0, -2.0)
assert.eq(5.1 - 7.1, -2.0)
assert.eq(5.5 - 7, -1.5)
assert.eq(5 - 7.5, -2.5)
assert.eq(0.0 - 1.0, -1.0)
# multiplication
assert.eq(5.0 * 7.0, 35.0)
assert.eq(5.5 * 2.5, 13.75)
assert.eq(5.5 * 7, 38.5)
assert.eq(5 * 7.1, 35.5)
# real division (like Python 3)
# The / operator is available only when the 'fp' dialect option is enabled.
assert.eq(100.0 / 8.0, 12.5)
assert.eq(100.0 / -8.0, -12.5)
assert.eq(-100.0 / 8.0, -12.5)
assert.eq(-100.0 / -8.0, 12.5)
assert.eq(98.0 / 8.0, 12.25)
assert.eq(98.0 / -8.0, -12.25)
assert.eq(-98.0 / 8.0, -12.25)
assert.eq(-98.0 / -8.0, 12.25)
assert.eq(2.5 / 2.0, 1.25)
assert.eq(2.5 / 2, 1.25)
assert.eq(5 / 4.0, 1.25)
assert.eq(5 / 4, 1.25)
assert.fails(lambda: 1.0 / 0, "floating-point division by zero")
assert.fails(lambda: 1.0 / 0.0, "floating-point division by zero")
assert.fails(lambda: 1 / 0.0, "floating-point division by zero")
# floored division
assert.eq(100.0 // 8.0, 12.0)
assert.eq(100.0 // -8.0, -13.0)
assert.eq(-100.0 // 8.0, -13.0)
assert.eq(-100.0 // -8.0, 12.0)
assert.eq(98.0 // 8.0, 12.0)
assert.eq(98.0 // -8.0, -13.0)
assert.eq(-98.0 // 8.0, -13.0)
assert.eq(-98.0 // -8.0, 12.0)
assert.eq(2.5 // 2.0, 1.0)
assert.eq(2.5 // 2, 1.0)
assert.eq(5 // 4.0, 1.0)
assert.eq(5 // 4, 1)
assert.eq(type(5 // 4), "int")
assert.fails(lambda: 1.0 // 0, "floored division by zero")
assert.fails(lambda: 1.0 // 0.0, "floored division by zero")
assert.fails(lambda: 1 // 0.0, "floored division by zero")
# remainder
assert.eq(100.0 % 8.0, 4.0)
assert.eq(100.0 % -8.0, -4.0)
assert.eq(-100.0 % 8.0, 4.0)
assert.eq(-100.0 % -8.0, -4.0)
assert.eq(98.0 % 8.0, 2.0)
assert.eq(98.0 % -8.0, -6.0)
assert.eq(-98.0 % 8.0, 6.0)
assert.eq(-98.0 % -8.0, -2.0)
assert.eq(2.5 % 2.0, 0.5)
assert.eq(2.5 % 2, 0.5)
assert.eq(5 % 4.0, 1.0)
assert.fails(lambda: 1.0 % 0, "floating-point modulo by zero")
assert.fails(lambda: 1.0 % 0.0, "floating-point modulo by zero")
assert.fails(lambda: 1 % 0.0, "floating-point modulo by zero")
# floats cannot be used as indices, even if integral
assert.fails(lambda: "abc"[1.0], "want int")
assert.fails(lambda: ["A", "B", "C"].insert(1.0, "D"), "want int")
assert.fails(lambda: range(3)[1.0], "got float, want int")
# -- comparisons --
# NaN
assert.true(nan == nan) # \
assert.true(nan >= nan) # unlike Python
assert.true(nan <= nan) # /
assert.true(not (nan > nan))
assert.true(not (nan < nan))
assert.true(not (nan != nan)) # unlike Python
# Sort is stable: 0.0 and -0.0 are equal, but they are not permuted.
# Similarly 1 and 1.0.
assert.eq(
str(sorted([inf, neginf, nan, 1e300, -1e300, 1.0, -1.0, 1, -1, 1e-300, -1e-300, 0, 0.0, negzero, 1e-300, -1e-300])),
"[-inf, -1e+300, -1.0, -1, -1e-300, -1e-300, 0, 0.0, -0.0, 1e-300, 1e-300, 1.0, 1, 1e+300, +inf, nan]")
# Sort is stable, and its result contains no adjacent x, y such that y > x.
# Note: Python's reverse sort is unstable; see https://bugs.python.org/issue36095.
assert.eq(str(sorted([7, 3, nan, 1, 9])), "[1, 3, 7, 9, nan]")
assert.eq(str(sorted([7, 3, nan, 1, 9], reverse=True)), "[nan, 9, 7, 3, 1]")
# All NaN values compare equal. (Identical objects compare equal.)
nandict = {nan: 1}
nandict[nan] = 2
assert.eq(len(nandict), 1) # (same as Python)
assert.eq(nandict[nan], 2) # (same as Python)
assert.fails(lambda: {nan: 1, nan: 2}, "duplicate key: nan")
nandict[float('nan')] = 3 # a distinct NaN object
assert.eq(str(nandict), "{nan: 3}") # (Python: {nan: 2, nan: 3})
assert.eq(str({inf: 1, neginf: 2}), "{+inf: 1, -inf: 2}")
# zero
assert.eq(0.0, negzero)
# inf
assert.eq(+inf / +inf, nan)
assert.eq(+inf / -inf, nan)
assert.eq(-inf / +inf, nan)
assert.eq(0.0 / +inf, 0.0)
assert.eq(0.0 / -inf, 0.0)
assert.true(inf > -inf)
assert.eq(inf, -neginf)
# TODO(adonovan): assert inf > any finite number, etc.
# negative zero
negz = -0
assert.eq(negz, 0)
# min/max ordering with NaN (the greatest float value)
assert.eq(max([1, nan, 3]), nan)
assert.eq(max([nan, 2, 3]), nan)
assert.eq(min([1, nan, 3]), 1)
assert.eq(min([nan, 2, 3]), 2)
# float/float comparisons
fltmax = 1.7976931348623157e+308 # approx
fltmin = 4.9406564584124654e-324 # approx
assert.lt(-inf, -fltmax)
assert.lt(-fltmax, -1.0)
assert.lt(-1.0, -fltmin)
assert.lt(-fltmin, 0.0)
assert.lt(0, fltmin)
assert.lt(fltmin, 1.0)
assert.lt(1.0, fltmax)
assert.lt(fltmax, inf)
# int/float comparisons
assert.eq(0, 0.0)
assert.eq(1, 1.0)
assert.eq(-1, -1.0)
assert.ne(-1, -1.0 + 1e-7)
assert.lt(-2, -2 + 1e-15)
# int conversion (rounds towards zero)
assert.eq(int(100.1), 100)
assert.eq(int(100.0), 100)
assert.eq(int(99.9), 99)
assert.eq(int(-99.9), -99)
assert.eq(int(-100.0), -100)
assert.eq(int(-100.1), -100)
assert.eq(int(1e100), int("10000000000000000159028911097599180468360808563945281389781327557747838772170381060813469985856815104"))
assert.fails(lambda: int(inf), "cannot convert.*infinity")
assert.fails(lambda: int(nan), "cannot convert.*NaN")
# -- float() function --
assert.eq(float(), 0.0)
# float(bool)
assert.eq(float(False), 0.0)
assert.eq(float(True), 1.0)
# float(int)
assert.eq(float(0), 0.0)
assert.eq(float(1), 1.0)
assert.eq(float(123), 123.0)
assert.eq(float(123 * 1000000 * 1000000 * 1000000 * 1000000 * 1000000), 1.23e+32)
# float(float)
assert.eq(float(1.1), 1.1)
assert.fails(lambda: float(None), "want number or string")
assert.ne(False, 0.0) # differs from Python
assert.ne(True, 1.0)
# float(string)
assert.eq(float("1.1"), 1.1)
assert.fails(lambda: float("1.1abc"), "invalid float literal")
assert.fails(lambda: float("1e100.0"), "invalid float literal")
assert.fails(lambda: float("1e1000"), "floating-point number too large")
assert.eq(float("-1.1"), -1.1)
assert.eq(float("+1.1"), +1.1)
assert.eq(float("+Inf"), inf)
assert.eq(float("-Inf"), neginf)
assert.eq(float("NaN"), nan)
assert.eq(float("NaN"), nan)
assert.eq(float("+NAN"), nan)
assert.eq(float("-nan"), nan)
assert.eq(str(float("Inf")), "+inf")
assert.eq(str(float("+INF")), "+inf")
assert.eq(str(float("-inf")), "-inf")
assert.eq(str(float("+InFiniTy")), "+inf")
assert.eq(str(float("-iNFiniTy")), "-inf")
assert.fails(lambda: float("one point two"), "invalid float literal: one point two")
assert.fails(lambda: float("1.2.3"), "invalid float literal: 1.2.3")
assert.fails(lambda: float(123 << 500 << 500 << 50), "int too large to convert to float")
assert.fails(lambda: float(-123 << 500 << 500 << 50), "int too large to convert to float")
assert.fails(lambda: float(str(-123 << 500 << 500 << 50)), "floating-point number too large")
# -- implicit float(int) conversions --
assert.fails(lambda: (1<<500<<500<<500) + 0.0, "int too large to convert to float")
assert.fails(lambda: 0.0 + (1<<500<<500<<500), "int too large to convert to float")
assert.fails(lambda: (1<<500<<500<<500) - 0.0, "int too large to convert to float")
assert.fails(lambda: 0.0 - (1<<500<<500<<500), "int too large to convert to float")
assert.fails(lambda: (1<<500<<500<<500) * 1.0, "int too large to convert to float")
assert.fails(lambda: 1.0 * (1<<500<<500<<500), "int too large to convert to float")
assert.fails(lambda: (1<<500<<500<<500) / 1.0, "int too large to convert to float")
assert.fails(lambda: 1.0 / (1<<500<<500<<500), "int too large to convert to float")
assert.fails(lambda: (1<<500<<500<<500) // 1.0, "int too large to convert to float")
assert.fails(lambda: 1.0 // (1<<500<<500<<500), "int too large to convert to float")
assert.fails(lambda: (1<<500<<500<<500) % 1.0, "int too large to convert to float")
assert.fails(lambda: 1.0 % (1<<500<<500<<500), "int too large to convert to float")
# -- int function --
assert.eq(int(0.0), 0)
assert.eq(int(1.0), 1)
assert.eq(int(1.1), 1)
assert.eq(int(0.9), 0)
assert.eq(int(-1.1), -1.0)
assert.eq(int(-1.0), -1.0)
assert.eq(int(-0.9), 0.0)
assert.eq(int(1.23e+32), 123000000000000004979083645550592)
assert.eq(int(-1.23e-32), 0)
assert.eq(int(1.23e-32), 0)
assert.fails(lambda: int(float("+Inf")), "cannot convert float infinity to integer")
assert.fails(lambda: int(float("-Inf")), "cannot convert float infinity to integer")
assert.fails(lambda: int(float("NaN")), "cannot convert float NaN to integer")
# hash
# Check that equal float and int values have the same internal hash.
def checkhash():
for a in [1.23e100, 1.23e10, 1.23e1, 1.23,
1, 4294967295, 8589934591, 9223372036854775807]:
for b in [a, -a, 1/a, -1/a]:
f = float(b)
i = int(b)
if f == i:
fh = {f: None}
ih = {i: None}
if fh != ih:
assert.true(False, "{%v: None} != {%v: None}: hashes vary" % fh, ih)
checkhash()
# string formatting
# %d
assert.eq("%d" % 0, "0")
assert.eq("%d" % 0.0, "0")
assert.eq("%d" % 123, "123")
assert.eq("%d" % 123.0, "123")
assert.eq("%d" % 1.23e45, "1229999999999999973814869011019624571608236032")
# (see below for '%d' % NaN/Inf)
assert.eq("%d" % negzero, "0")
assert.fails(lambda: "%d" % float("NaN"), "cannot convert float NaN to integer")
assert.fails(lambda: "%d" % float("+Inf"), "cannot convert float infinity to integer")
assert.fails(lambda: "%d" % float("-Inf"), "cannot convert float infinity to integer")
# %e
assert.eq("%e" % 0, "0.000000e+00")
assert.eq("%e" % 0.0, "0.000000e+00")
assert.eq("%e" % 123, "1.230000e+02")
assert.eq("%e" % 123.0, "1.230000e+02")
assert.eq("%e" % 1.23e45, "1.230000e+45")
assert.eq("%e" % -1.23e-45, "-1.230000e-45")
assert.eq("%e" % nan, "nan")
assert.eq("%e" % inf, "+inf")
assert.eq("%e" % neginf, "-inf")
assert.eq("%e" % negzero, "-0.000000e+00")
assert.fails(lambda: "%e" % "123", "requires float, not str")
# %f
assert.eq("%f" % 0, "0.000000")
assert.eq("%f" % 0.0, "0.000000")
assert.eq("%f" % 123, "123.000000")
assert.eq("%f" % 123.0, "123.000000")
# Note: Starlark/Java emits 1230000000000000000000000000000000000000000000.000000. Why?
assert.eq("%f" % 1.23e45, "1229999999999999973814869011019624571608236032.000000")
assert.eq("%f" % -1.23e-45, "-0.000000")
assert.eq("%f" % nan, "nan")
assert.eq("%f" % inf, "+inf")
assert.eq("%f" % neginf, "-inf")
assert.eq("%f" % negzero, "-0.000000")
assert.fails(lambda: "%f" % "123", "requires float, not str")
# %g
assert.eq("%g" % 0, "0.0")
assert.eq("%g" % 0.0, "0.0")
assert.eq("%g" % 123, "123.0")
assert.eq("%g" % 123.0, "123.0")
assert.eq("%g" % 1.110, "1.11")
assert.eq("%g" % 1e5, "100000.0")
assert.eq("%g" % 1e6, "1e+06") # Note: threshold of scientific notation is 1e17 in Starlark/Java
assert.eq("%g" % 1.23e45, "1.23e+45")
assert.eq("%g" % -1.23e-45, "-1.23e-45")
assert.eq("%g" % nan, "nan")
assert.eq("%g" % inf, "+inf")
assert.eq("%g" % neginf, "-inf")
assert.eq("%g" % negzero, "-0.0")
# str uses %g
assert.eq(str(0.0), "0.0")
assert.eq(str(123.0), "123.0")
assert.eq(str(1.23e45), "1.23e+45")
assert.eq(str(-1.23e-45), "-1.23e-45")
assert.eq(str(nan), "nan")
assert.eq(str(inf), "+inf")
assert.eq(str(neginf), "-inf")
assert.eq(str(negzero), "-0.0")
assert.fails(lambda: "%g" % "123", "requires float, not str")
i0 = 1
f0 = 1.0
assert.eq(type(i0), "int")
assert.eq(type(f0), "float")
ops = {
'+': lambda x, y: x + y,
'-': lambda x, y: x - y,
'*': lambda x, y: x * y,
'/': lambda x, y: x / y,
'//': lambda x, y: x // y,
'%': lambda x, y: x % y,
}
# Check that if either argument is a float, so too is the result.
def checktypes():
want = set("""
int + int = int
int + float = float
float + int = float
float + float = float
int - int = int
int - float = float
float - int = float
float - float = float
int * int = int
int * float = float
float * int = float
float * float = float
int / int = float
int / float = float
float / int = float
float / float = float
int // int = int
int // float = float
float // int = float
float // float = float
int % int = int
int % float = float
float % int = float
float % float = float
"""[1:].splitlines())
for opname in ("+", "-", "*", "/", "%"):
for x in [i0, f0]:
for y in [i0, f0]:
op = ops[opname]
got = "%s %s %s = %s" % (type(x), opname, type(y), type(op(x, y)))
assert.contains(want, got)
checktypes()
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/function.star
Vendored
+329
View File
@@ -0,0 +1,329 @@
# Tests of Starlark 'function'
# option:set
# TODO(adonovan):
# - add some introspection functions for looking at function values
# and test that functions have correct position, free vars, names of locals, etc.
# - move the hard-coded tests of parameter passing from eval_test.go to here.
load("assert.star", "assert", "freeze")
# Test lexical scope and closures:
def outer(x):
def inner(y):
return x + x + y # multiple occurrences of x should create only 1 freevar
return inner
z = outer(3)
assert.eq(z(5), 11)
assert.eq(z(7), 13)
z2 = outer(4)
assert.eq(z2(5), 13)
assert.eq(z2(7), 15)
assert.eq(z(5), 11)
assert.eq(z(7), 13)
# Function name
assert.eq(str(outer), '<function outer>')
assert.eq(str(z), '<function inner>')
assert.eq(str(str), '<built-in function str>')
assert.eq(str("".startswith), '<built-in method startswith of string value>')
# Stateful closure
def squares():
x = [0]
def f():
x[0] += 1
return x[0] * x[0]
return f
sq = squares()
assert.eq(sq(), 1)
assert.eq(sq(), 4)
assert.eq(sq(), 9)
assert.eq(sq(), 16)
# Freezing a closure
sq2 = freeze(sq)
assert.fails(sq2, "frozen list")
# recursion detection, simple
def fib(x):
if x < 2:
return x
return fib(x-2) + fib(x-1)
assert.fails(lambda: fib(10), "function fib called recursively")
# recursion detection, advanced
#
# A simplistic recursion check that looks for repeated calls to the
# same function value will not detect recursion using the Y
# combinator, which creates a new closure at each step of the
# recursion. To truly prohibit recursion, the dynamic check must look
# for repeated calls of the same syntactic function body.
Y = lambda f: (lambda x: x(x))(lambda y: f(lambda *args: y(y)(*args)))
fibgen = lambda fib: lambda x: (x if x<2 else fib(x-1)+fib(x-2))
fib2 = Y(fibgen)
assert.fails(lambda: [fib2(x) for x in range(10)], "function lambda called recursively")
# However, this stricter check outlaws many useful programs
# that are still bounded, and creates a hazard because
# helper functions such as map below cannot be used to
# call functions that themselves use map:
def map(f, seq): return [f(x) for x in seq]
def double(x): return x+x
assert.eq(map(double, [1, 2, 3]), [2, 4, 6])
assert.eq(map(double, ["a", "b", "c"]), ["aa", "bb", "cc"])
def mapdouble(x): return map(double, x)
assert.fails(lambda: map(mapdouble, ([1, 2, 3], ["a", "b", "c"])),
'function map called recursively')
# With the -recursion option it would yield [[2, 4, 6], ["aa", "bb", "cc"]].
# call of function not through its name
# (regression test for parsing suffixes of primary expressions)
hf = hasfields()
hf.x = [len]
assert.eq(hf.x[0]("abc"), 3)
def f():
return lambda: 1
assert.eq(f()(), 1)
assert.eq(["abc"][0][0].upper(), "A")
# functions may be recursively defined,
# so long as they don't dynamically recur.
calls = []
def yin(x):
calls.append("yin")
if x:
yang(False)
def yang(x):
calls.append("yang")
if x:
yin(False)
yin(True)
assert.eq(calls, ["yin", "yang"])
calls.clear()
yang(True)
assert.eq(calls, ["yang", "yin"])
# builtin_function_or_method use identity equivalence.
closures = set(["".count for _ in range(10)])
assert.eq(len(closures), 10)
---
# Default values of function parameters are mutable.
load("assert.star", "assert", "freeze")
def f(x=[0]):
return x
assert.eq(f(), [0])
f().append(1)
assert.eq(f(), [0, 1])
# Freezing a function value freezes its parameter defaults.
freeze(f)
assert.fails(lambda: f().append(2), "cannot append to frozen list")
---
# This is a well known corner case of parsing in Python.
load("assert.star", "assert")
f = lambda x: 1 if x else 0
assert.eq(f(True), 1)
assert.eq(f(False), 0)
x = True
f2 = (lambda x: 1) if x else 0
assert.eq(f2(123), 1)
tf = lambda: True, lambda: False
assert.true(tf[0]())
assert.true(not tf[1]())
---
# Missing parameters are correctly reported
# in functions of more than 64 parameters.
# (This tests a corner case of the implementation:
# we avoid a map allocation for <64 parameters)
load("assert.star", "assert")
def f(a, b, c, d, e, f, g, h,
i, j, k, l, m, n, o, p,
q, r, s, t, u, v, w, x,
y, z, A, B, C, D, E, F,
G, H, I, J, K, L, M, N,
O, P, Q, R, S, T, U, V,
W, X, Y, Z, aa, bb, cc, dd,
ee, ff, gg, hh, ii, jj, kk, ll,
mm):
pass
assert.fails(lambda: f(
1, 2, 3, 4, 5, 6, 7, 8,
9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, 22, 23, 24,
25, 26, 27, 28, 29, 30, 31, 32,
33, 34, 35, 36, 37, 38, 39, 40,
41, 42, 43, 44, 45, 46, 47, 48,
49, 50, 51, 52, 53, 54, 55, 56,
57, 58, 59, 60, 61, 62, 63, 64), "missing 1 argument \\(mm\\)")
assert.fails(lambda: f(
1, 2, 3, 4, 5, 6, 7, 8,
9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, 22, 23, 24,
25, 26, 27, 28, 29, 30, 31, 32,
33, 34, 35, 36, 37, 38, 39, 40,
41, 42, 43, 44, 45, 46, 47, 48,
49, 50, 51, 52, 53, 54, 55, 56,
57, 58, 59, 60, 61, 62, 63, 64, 65,
mm = 100), 'multiple values for parameter "mm"')
---
# Regression test for github.com/google/starlark-go/issues/21,
# which concerns dynamic checks.
# Related: https://github.com/bazelbuild/starlark/issues/21,
# which concerns static checks.
load("assert.star", "assert")
def f(*args, **kwargs):
return args, kwargs
assert.eq(f(x=1, y=2), ((), {"x": 1, "y": 2}))
assert.fails(lambda: f(x=1, **dict(x=2)), 'multiple values for parameter "x"')
def g(x, y):
return x, y
assert.eq(g(1, y=2), (1, 2))
assert.fails(lambda: g(1, y=2, **{'y': 3}), 'multiple values for parameter "y"')
---
# Regression test for a bug in CALL_VAR_KW.
load("assert.star", "assert")
def f(a, b, x, y):
return a+b+x+y
assert.eq(f(*("a", "b"), **dict(y="y", x="x")) + ".", 'abxy.')
---
# Order of evaluation of function arguments.
# Regression test for github.com/google/skylark/issues/135.
load("assert.star", "assert")
r = []
def id(x):
r.append(x)
return x
def f(*args, **kwargs):
return (args, kwargs)
y = f(id(1), id(2), x=id(3), *[id(4)], **dict(z=id(5)))
assert.eq(y, ((1, 2, 4), dict(x=3, z=5)))
# This matches Python2 and Starlark-in-Java, but not Python3 [1 2 4 3 6].
# *args and *kwargs are evaluated last.
# (Python[23] also allows keyword arguments after *args.)
# See github.com/bazelbuild/starlark#13 for spec change.
assert.eq(r, [1, 2, 3, 4, 5])
---
# option:recursion
# See github.com/bazelbuild/starlark#170
load("assert.star", "assert")
def a():
list = []
def b(n):
list.append(n)
if n > 0:
b(n - 1) # recursive reference to b
b(3)
return list
assert.eq(a(), [3, 2, 1, 0])
def c():
list = []
x = 1
def d():
list.append(x) # this use of x observes both assignments
d()
x = 2
d()
return list
assert.eq(c(), [1, 2])
def e():
def f():
return x # forward reference ok: x is a closure cell
x = 1
return f()
assert.eq(e(), 1)
---
load("assert.star", "assert")
def e():
x = 1
def f():
print(x) # this reference to x fails
x = 3 # because this assignment makes x local to f
f()
assert.fails(e, "local variable x referenced before assignment")
def f():
def inner():
return x
if False:
x = 0
return x # fails (x is an uninitialized cell of this function)
assert.fails(f, "local variable x referenced before assignment")
def g():
def inner():
return x # fails (x is an uninitialized cell of the enclosing function)
if False:
x = 0
return inner()
assert.fails(g, "local variable x referenced before assignment")
---
# A trailing comma is allowed in any function definition or call.
# This reduces the need to edit neighboring lines when editing defs
# or calls splayed across multiple lines.
def a(x,): pass
def b(x, y=None, ): pass
def c(x, y=None, *args, ): pass
def d(x, y=None, *args, z=None, ): pass
def e(x, y=None, *args, z=None, **kwargs, ): pass
a(1,)
b(1, y=2, )
#c(1, *[], )
#d(1, *[], z=None, )
#e(1, *[], z=None, *{}, )
---
# Unpack provides spell check for argument names.
load("assert.star", "assert")
assert.fails(lambda: min([], keg=1), ".+did you mean key\\?")
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/function_param.star
Vendored
+10
View File
@@ -0,0 +1,10 @@
# Functions used in starlark_test.TestParamDefault().
def all_required(a, b, c): pass
def all_opt(a="a", b=None, c=""): pass
def mix_required_opt(a, b, c="c", d="d"): pass
def with_varargs(a, b="b", *args): pass
def with_varargs_kwonly(a, b="b", *args, c="c", d): pass
def with_kwonly(a, b="b", *, c="c", d): pass
def with_kwargs(a, b="b", c="c", **kwargs): pass
def with_varargs_kwonly_kwargs(a, b="b", *args, c="c", d, e="e", **kwargs): pass
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/int.star
Vendored
+258
View File
@@ -0,0 +1,258 @@
# Tests of Starlark 'int'
load("assert.star", "assert")
# basic arithmetic
assert.eq(0 - 1, -1)
assert.eq(0 + 1, +1)
assert.eq(1 + 1, 2)
assert.eq(5 + 7, 12)
assert.eq(5 * 7, 35)
assert.eq(5 - 7, -2)
# int boundaries
maxint64 = (1 << 63) - 1
minint64 = -1 << 63
maxint32 = (1 << 31) - 1
minint32 = -1 << 31
assert.eq(maxint64, 9223372036854775807)
assert.eq(minint64, -9223372036854775808)
assert.eq(maxint32, 2147483647)
assert.eq(minint32, -2147483648)
# truth
def truth():
assert.true(not 0)
for m in [1, maxint32]: # Test small/big ranges
assert.true(123 * m)
assert.true(-1 * m)
truth()
# floored division
# (For real division, see float.star.)
def division():
for m in [1, maxint32]: # Test small/big ranges
assert.eq((100 * m) // (7 * m), 14)
assert.eq((100 * m) // (-7 * m), -15)
assert.eq((-100 * m) // (7 * m), -15) # NB: different from Go/Java
assert.eq((-100 * m) // (-7 * m), 14) # NB: different from Go/Java
assert.eq((98 * m) // (7 * m), 14)
assert.eq((98 * m) // (-7 * m), -14)
assert.eq((-98 * m) // (7 * m), -14)
assert.eq((-98 * m) // (-7 * m), 14)
division()
# remainder
def remainder():
for m in [1, maxint32]: # Test small/big ranges
assert.eq((100 * m) % (7 * m), 2 * m)
assert.eq((100 * m) % (-7 * m), -5 * m) # NB: different from Go/Java
assert.eq((-100 * m) % (7 * m), 5 * m) # NB: different from Go/Java
assert.eq((-100 * m) % (-7 * m), -2 * m)
assert.eq((98 * m) % (7 * m), 0)
assert.eq((98 * m) % (-7 * m), 0)
assert.eq((-98 * m) % (7 * m), 0)
assert.eq((-98 * m) % (-7 * m), 0)
remainder()
# compound assignment
def compound():
x = 1
x += 1
assert.eq(x, 2)
x -= 3
assert.eq(x, -1)
x *= 39
assert.eq(x, -39)
x //= 4
assert.eq(x, -10)
x /= -2
assert.eq(x, 5)
x %= 3
assert.eq(x, 2)
x = 2
x &= 1
assert.eq(x, 0)
x |= 2
assert.eq(x, 2)
x ^= 3
assert.eq(x, 1)
x <<= 2
assert.eq(x, 4)
x >>= 2
assert.eq(x, 1)
compound()
# int conversion
# See float.star for float-to-int conversions.
# We follow Python 3 here, but I can't see the method in its madness.
# int from bool/int/float
assert.fails(int, "missing argument") # int()
assert.eq(int(False), 0)
assert.eq(int(True), 1)
assert.eq(int(3), 3)
assert.eq(int(3.1), 3)
assert.fails(lambda: int(3, base = 10), "non-string with explicit base")
assert.fails(lambda: int(True, 10), "non-string with explicit base")
# int from string, base implicitly 10
assert.eq(int("100000000000000000000"), 10000000000 * 10000000000)
assert.eq(int("-100000000000000000000"), -10000000000 * 10000000000)
assert.eq(int("123"), 123)
assert.eq(int("-123"), -123)
assert.eq(int("0123"), 123) # not octal
assert.eq(int("-0123"), -123)
assert.fails(lambda: int("0x12"), "invalid literal with base 10")
assert.fails(lambda: int("-0x12"), "invalid literal with base 10")
assert.fails(lambda: int("0o123"), "invalid literal.*base 10")
assert.fails(lambda: int("-0o123"), "invalid literal.*base 10")
# int from string, explicit base
assert.eq(int("0"), 0)
assert.eq(int("00"), 0)
assert.eq(int("0", base = 10), 0)
assert.eq(int("00", base = 10), 0)
assert.eq(int("0", base = 8), 0)
assert.eq(int("00", base = 8), 0)
assert.eq(int("-0"), 0)
assert.eq(int("-00"), 0)
assert.eq(int("-0", base = 10), 0)
assert.eq(int("-00", base = 10), 0)
assert.eq(int("-0", base = 8), 0)
assert.eq(int("-00", base = 8), 0)
assert.eq(int("+0"), 0)
assert.eq(int("+00"), 0)
assert.eq(int("+0", base = 10), 0)
assert.eq(int("+00", base = 10), 0)
assert.eq(int("+0", base = 8), 0)
assert.eq(int("+00", base = 8), 0)
assert.eq(int("11", base = 9), 10)
assert.eq(int("-11", base = 9), -10)
assert.eq(int("10011", base = 2), 19)
assert.eq(int("-10011", base = 2), -19)
assert.eq(int("123", 8), 83)
assert.eq(int("-123", 8), -83)
assert.eq(int("0123", 8), 83) # redundant zeros permitted
assert.eq(int("-0123", 8), -83)
assert.eq(int("00123", 8), 83)
assert.eq(int("-00123", 8), -83)
assert.eq(int("0o123", 8), 83)
assert.eq(int("-0o123", 8), -83)
assert.eq(int("123", 7), 66) # 1*7*7 + 2*7 + 3
assert.eq(int("-123", 7), -66)
assert.eq(int("12", 16), 18)
assert.eq(int("-12", 16), -18)
assert.eq(int("0x12", 16), 18)
assert.eq(int("-0x12", 16), -18)
assert.eq(0x1000000000000001 * 0x1000000000000001, 0x1000000000000002000000000000001)
assert.eq(int("1010", 2), 10)
assert.eq(int("111111101", 2), 509)
assert.eq(int("0b0101", 0), 5)
assert.eq(int("0b0101", 2), 5) # prefix is redundant with explicit base
assert.eq(int("0b00000", 0), 0)
assert.eq(1111111111111111 * 1111111111111111, 1234567901234567654320987654321)
assert.fails(lambda: int("0x123", 8), "invalid literal.*base 8")
assert.fails(lambda: int("-0x123", 8), "invalid literal.*base 8")
assert.fails(lambda: int("0o123", 16), "invalid literal.*base 16")
assert.fails(lambda: int("-0o123", 16), "invalid literal.*base 16")
assert.fails(lambda: int("0x110", 2), "invalid literal.*base 2")
# Base prefix is honored only if base=0, or if the prefix matches the explicit base.
# See https://github.com/google/starlark-go/issues/337
assert.fails(lambda: int("0b0"), "invalid literal.*base 10")
assert.eq(int("0b0", 0), 0)
assert.eq(int("0b0", 2), 0)
assert.eq(int("0b0", 16), 0xb0)
assert.eq(int("0x0b0", 16), 0xb0)
assert.eq(int("0x0b0", 0), 0xb0)
assert.eq(int("0x0b0101", 16), 0x0b0101)
# int from string, auto detect base
assert.eq(int("123", 0), 123)
assert.eq(int("+123", 0), +123)
assert.eq(int("-123", 0), -123)
assert.eq(int("0x12", 0), 18)
assert.eq(int("+0x12", 0), +18)
assert.eq(int("-0x12", 0), -18)
assert.eq(int("0o123", 0), 83)
assert.eq(int("+0o123", 0), +83)
assert.eq(int("-0o123", 0), -83)
assert.fails(lambda: int("0123", 0), "invalid literal.*base 0") # valid in Python 2.7
assert.fails(lambda: int("-0123", 0), "invalid literal.*base 0")
# github.com/google/starlark-go/issues/108
assert.fails(lambda: int("0Oxa", 8), "invalid literal with base 8: 0Oxa")
# follow-on bugs to issue 108
assert.fails(lambda: int("--4"), "invalid literal with base 10: --4")
assert.fails(lambda: int("++4"), "invalid literal with base 10: \\+\\+4")
assert.fails(lambda: int("+-4"), "invalid literal with base 10: \\+-4")
assert.fails(lambda: int("0x-4", 16), "invalid literal with base 16: 0x-4")
# bitwise union (int|int), intersection (int&int), XOR (int^int), unary not (~int),
# left shift (int<<int), and right shift (int>>int).
# TODO(adonovan): this is not yet in the Starlark spec,
# but there is consensus that it should be.
assert.eq(1 | 2, 3)
assert.eq(3 | 6, 7)
assert.eq((1 | 2) & (2 | 4), 2)
assert.eq(1 ^ 2, 3)
assert.eq(2 ^ 2, 0)
assert.eq(1 | 0 ^ 1, 1) # check | and ^ operators precedence
assert.eq(~1, -2)
assert.eq(~(-2), 1)
assert.eq(~0, -1)
assert.eq(1 << 2, 4)
assert.eq(2 >> 1, 1)
assert.fails(lambda: 2 << -1, "negative shift count")
assert.fails(lambda: 1 << 512, "shift count too large")
# comparisons
# TODO(adonovan): test: < > == != etc
def comparisons():
for m in [1, maxint32 / 2, maxint32]: # Test small/big ranges
assert.lt(-2 * m, -1 * m)
assert.lt(-1 * m, 0 * m)
assert.lt(0 * m, 1 * m)
assert.lt(1 * m, 2 * m)
assert.true(2 * m >= 2 * m)
assert.true(2 * m > 1 * m)
assert.true(1 * m >= 1 * m)
assert.true(1 * m > 0 * m)
assert.true(0 * m >= 0 * m)
assert.true(0 * m > -1 * m)
assert.true(-1 * m >= -1 * m)
assert.true(-1 * m > -2 * m)
comparisons()
# precision
assert.eq(str(maxint64), "9223372036854775807")
assert.eq(str(maxint64 + 1), "9223372036854775808")
assert.eq(str(minint64), "-9223372036854775808")
assert.eq(str(minint64 - 1), "-9223372036854775809")
assert.eq(str(minint64 * minint64), "85070591730234615865843651857942052864")
assert.eq(str(maxint32 + 1), "2147483648")
assert.eq(str(minint32 - 1), "-2147483649")
assert.eq(str(minint32 * minint32), "4611686018427387904")
assert.eq(str(minint32 | maxint32), "-1")
assert.eq(str(minint32 & minint32), "-2147483648")
assert.eq(str(minint32 ^ maxint32), "-1")
assert.eq(str(minint32 // -1), "2147483648")
# string formatting
assert.eq("%o %x %d" % (0o755, 0xDEADBEEF, 42), "755 deadbeef 42")
nums = [-95, -1, 0, +1, +95]
assert.eq(" ".join(["%o" % x for x in nums]), "-137 -1 0 1 137")
assert.eq(" ".join(["%d" % x for x in nums]), "-95 -1 0 1 95")
assert.eq(" ".join(["%i" % x for x in nums]), "-95 -1 0 1 95")
assert.eq(" ".join(["%x" % x for x in nums]), "-5f -1 0 1 5f")
assert.eq(" ".join(["%X" % x for x in nums]), "-5F -1 0 1 5F")
assert.eq("%o %x %d" % (123, 123, 123), "173 7b 123")
assert.eq("%o %x %d" % (123.1, 123.1, 123.1), "173 7b 123") # non-int operands are acceptable
assert.fails(lambda: "%d" % True, "cannot convert bool to int")
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/json.star
Vendored
+173
View File
@@ -0,0 +1,173 @@
# Tests of json module.
load("assert.star", "assert")
load("json.star", "json")
assert.eq(dir(json), ["decode", "encode", "indent"])
# Some of these cases were inspired by github.com/nst/JSONTestSuite.
## json.encode
assert.eq(json.encode(None), "null")
assert.eq(json.encode(True), "true")
assert.eq(json.encode(False), "false")
assert.eq(json.encode(-123), "-123")
assert.eq(json.encode(12345*12345*12345*12345*12345*12345), "3539537889086624823140625")
assert.eq(json.encode(float(12345*12345*12345*12345*12345*12345)), "3.539537889086625e+24")
assert.eq(json.encode(12.345e67), "1.2345e+68")
assert.eq(json.encode("hello"), '"hello"')
assert.eq(json.encode([1, 2, 3]), "[1,2,3]")
assert.eq(json.encode((1, 2, 3)), "[1,2,3]")
assert.eq(json.encode(range(3)), "[0,1,2]") # a built-in iterable
assert.eq(json.encode(dict(x = 1, y = "two")), '{"x":1,"y":"two"}')
assert.eq(json.encode(dict(y = "two", x = 1)), '{"x":1,"y":"two"}') # key, not insertion, order
assert.eq(json.encode(struct(x = 1, y = "two")), '{"x":1,"y":"two"}') # a user-defined HasAttrs
assert.eq(json.encode("😹"[:1]), '"\\ufffd"') # invalid UTF-8 -> replacement char
def encode_error(expr, error):
assert.fails(lambda: json.encode(expr), error)
encode_error(float("NaN"), "json.encode: cannot encode non-finite float nan")
encode_error({1: "two"}, "dict has int key, want string")
encode_error(len, "cannot encode builtin_function_or_method as JSON")
encode_error(struct(x=[1, {"x": len}]), # nested failure
'in field .x: at list index 1: in dict key "x": cannot encode...')
encode_error(struct(x=[1, {"x": len}]), # nested failure
'in field .x: at list index 1: in dict key "x": cannot encode...')
encode_error({1: 2}, 'dict has int key, want string')
recursive_map = {}
recursive_map["r"] = recursive_map
encode_error(recursive_map, 'json.encode: in dict key "r": cycle in JSON structure')
recursive_list = []
recursive_list.append(recursive_list)
encode_error(recursive_list, 'json.encode: at list index 0: cycle in JSON structure')
recursive_tuple = (1, 2, [])
recursive_tuple[2].append(recursive_tuple)
encode_error(recursive_tuple, 'json.encode: at tuple index 2: at list index 0: cycle in JSON structure')
## json.decode
assert.eq(json.decode("null"), None)
assert.eq(json.decode("true"), True)
assert.eq(json.decode("false"), False)
assert.eq(json.decode("-123"), -123)
assert.eq(json.decode("-0"), -0)
assert.eq(json.decode("3539537889086624823140625"), 3539537889086624823140625)
assert.eq(json.decode("3539537889086624823140625.0"), float(3539537889086624823140625))
assert.eq(json.decode("3.539537889086625e+24"), 3.539537889086625e+24)
assert.eq(json.decode("0e+1"), 0)
assert.eq(json.decode("-0.0"), -0.0)
assert.eq(json.decode(
"-0.000000000000000000000000000000000000000000000000000000000000000000000000000001"),
-0.000000000000000000000000000000000000000000000000000000000000000000000000000001)
assert.eq(json.decode('[]'), [])
assert.eq(json.decode('[1]'), [1])
assert.eq(json.decode('[1,2,3]'), [1, 2, 3])
assert.eq(json.decode('{"one": 1, "two": 2}'), dict(one=1, two=2))
assert.eq(json.decode('{"foo\\u0000bar": 42}'), {"foo\x00bar": 42})
assert.eq(json.decode('"\\ud83d\\ude39\\ud83d\\udc8d"'), "😹💍")
assert.eq(json.decode('"\\u0123"'), 'ģ')
assert.eq(json.decode('"\x7f"'), "\x7f")
def decode_error(expr, error):
assert.fails(lambda: json.decode(expr), error)
decode_error('truefalse',
"json.decode: at offset 4, unexpected character 'f' after value")
decode_error('"abc', "unclosed string literal")
decode_error('"ab\\gc"', "invalid character 'g' in string escape code")
decode_error("'abc'", "unexpected character '\\\\''")
decode_error("1.2.3", "invalid number: 1.2.3")
decode_error("+1", "unexpected character '\\+'")
decode_error("-abc", "invalid number: -")
decode_error("-", "invalid number: -")
decode_error("-00", "invalid number: -00")
decode_error("00", "invalid number: 00")
decode_error("--1", "invalid number: --1")
decode_error("-+1", "invalid number: -\\+1")
decode_error("1e1e1", "invalid number: 1e1e1")
decode_error("0123", "invalid number: 0123")
decode_error("000.123", "invalid number: 000.123")
decode_error("-0123", "invalid number: -0123")
decode_error("-000.123", "invalid number: -000.123")
decode_error("0x123", "unexpected character 'x' after value")
decode_error('[1, 2 ', "unexpected end of file")
decode_error('[1, 2, ', "unexpected end of file")
decode_error('[1, 2, ]', "unexpected character ']'")
decode_error('[1, 2, }', "unexpected character '}'")
decode_error('[1, 2}', "got '}', want ',' or ']'")
decode_error('{"one": 1', "unexpected end of file")
decode_error('{"one" 1', "after object key, got '1', want ':'")
decode_error('{"one": 1 "two": 2', "in object, got '\"', want ',' or '}'")
decode_error('{"one": 1,', "unexpected end of file")
decode_error('{"one": 1, }', "unexpected character '}'")
decode_error('{"one": 1]', "in object, got ']', want ',' or '}'")
## json.decode with default specified
assert.eq(json.decode('{"valid": "json"}', default = "default value"), {"valid": "json"})
assert.eq(json.decode('{"valid": "json"}', "default value"), {"valid": "json"})
assert.eq(json.decode('{"invalid": "json"', default = "default value"), "default value")
assert.eq(json.decode('{"invalid": "json"', "default value"), "default value")
assert.eq(json.decode('{"invalid": "json"', default = None), None)
assert.eq(json.decode('{"invalid": "json"', None), None)
assert.fails(
lambda: json.decode(x = '{"invalid": "json"', default = "default value"),
"unexpected keyword argument x"
)
def codec(x):
return json.decode(json.encode(x))
# string round-tripping
strings = [
"😿", # U+1F63F CRYING_CAT_FACE
"🐱‍👤", # CAT FACE + ZERO WIDTH JOINER + BUST IN SILHOUETTE
]
assert.eq(codec(strings), strings)
# codepoints is a string with every 16-bit code point.
codepoints = ''.join(['%c' % c for c in range(65536)])
assert.eq(codec(codepoints), codepoints)
# number round-tripping
numbers = [
0, 1, -1, +1, 1.23e45, -1.23e-45,
3539537889086624823140625,
float(3539537889086624823140625),
]
assert.eq(codec(numbers), numbers)
## json.indent
s = json.encode(dict(x = 1, y = ["one", "two"]))
assert.eq(json.indent(s), '''{
"x": 1,
"y": [
"one",
"two"
]
}''')
assert.eq(json.decode(json.indent(s)), {"x": 1, "y": ["one", "two"]})
assert.eq(json.indent(s, prefix='', indent='–––'), '''{
–––"x": 1,
–––"y": [
––––––"one",
––––––"two"
–––]
}''')
assert.fails(lambda: json.indent("!@#$%^& this is not json"), 'invalid character')
---
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/list.star
Vendored
+276
View File
@@ -0,0 +1,276 @@
# Tests of Starlark 'list'
load("assert.star", "assert", "freeze")
# literals
assert.eq([], [])
assert.eq([1], [1])
assert.eq([1], [1])
assert.eq([1, 2], [1, 2])
assert.ne([1, 2, 3], [1, 2, 4])
# truth
assert.true([0])
assert.true(not [])
# indexing, x[i]
abc = list("abc".elems())
assert.fails(lambda: abc[-4], "list index -4 out of range \\[-3:2]")
assert.eq(abc[-3], "a")
assert.eq(abc[-2], "b")
assert.eq(abc[-1], "c")
assert.eq(abc[0], "a")
assert.eq(abc[1], "b")
assert.eq(abc[2], "c")
assert.fails(lambda: abc[3], "list index 3 out of range \\[-3:2]")
# x[i] = ...
x3 = [0, 1, 2]
x3[1] = 2
x3[2] += 3
assert.eq(x3, [0, 2, 5])
def f2():
x3[3] = 4
assert.fails(f2, "out of range")
freeze(x3)
def f3():
x3[0] = 0
assert.fails(f3, "cannot assign to element of frozen list")
assert.fails(x3.clear, "cannot clear frozen list")
# list + list
assert.eq([1, 2, 3] + [3, 4, 5], [1, 2, 3, 3, 4, 5])
assert.fails(lambda: [1, 2] + (3, 4), "unknown.*list \\+ tuple")
assert.fails(lambda: (1, 2) + [3, 4], "unknown.*tuple \\+ list")
# list * int, int * list
assert.eq(abc * 0, [])
assert.eq(abc * -1, [])
assert.eq(abc * 1, abc)
assert.eq(abc * 3, ["a", "b", "c", "a", "b", "c", "a", "b", "c"])
assert.eq(0 * abc, [])
assert.eq(-1 * abc, [])
assert.eq(1 * abc, abc)
assert.eq(3 * abc, ["a", "b", "c", "a", "b", "c", "a", "b", "c"])
# list comprehensions
assert.eq([2 * x for x in [1, 2, 3]], [2, 4, 6])
assert.eq([2 * x for x in [1, 2, 3] if x > 1], [4, 6])
assert.eq(
[(x, y) for x in [1, 2] for y in [3, 4]],
[(1, 3), (1, 4), (2, 3), (2, 4)],
)
assert.eq([(x, y) for x in [1, 2] if x == 2 for y in [3, 4]], [(2, 3), (2, 4)])
assert.eq([2 * x for x in (1, 2, 3)], [2, 4, 6])
assert.eq([x for x in "abc".elems()], ["a", "b", "c"])
assert.eq([x for x in {"a": 1, "b": 2}], ["a", "b"])
assert.eq([(y, x) for x, y in {1: 2, 3: 4}.items()], [(2, 1), (4, 3)])
# corner cases of parsing:
assert.eq([x for x in range(12) if x % 2 == 0 if x % 3 == 0], [0, 6])
assert.eq([x for x in [1, 2] if lambda: None], [1, 2])
assert.eq([x for x in [1, 2] if (lambda: 3 if True else 4)], [1, 2])
# list function
assert.eq(list(), [])
assert.eq(list("ab".elems()), ["a", "b"])
# A list comprehension defines a separate lexical block,
# whether at top-level...
a = [1, 2]
b = [a for a in [3, 4]]
assert.eq(a, [1, 2])
assert.eq(b, [3, 4])
# ...or local to a function.
def listcompblock():
c = [1, 2]
d = [c for c in [3, 4]]
assert.eq(c, [1, 2])
assert.eq(d, [3, 4])
listcompblock()
# list.pop
x4 = [1, 2, 3, 4, 5]
assert.fails(lambda: x4.pop(-6), "index -6 out of range \\[-5:4]")
assert.fails(lambda: x4.pop(6), "index 6 out of range \\[-5:4]")
assert.eq(x4.pop(), 5)
assert.eq(x4, [1, 2, 3, 4])
assert.eq(x4.pop(1), 2)
assert.eq(x4, [1, 3, 4])
assert.eq(x4.pop(0), 1)
assert.eq(x4, [3, 4])
assert.eq(x4.pop(-2), 3)
assert.eq(x4, [4])
assert.eq(x4.pop(-1), 4)
assert.eq(x4, [])
# TODO(adonovan): test uses of list as sequence
# (for loop, comprehension, library functions).
# x += y for lists is equivalent to x.extend(y).
# y may be a sequence.
# TODO: Test that side-effects of 'x' occur only once.
def list_extend():
a = [1, 2, 3]
b = a
a = a + [4] # creates a new list
assert.eq(a, [1, 2, 3, 4])
assert.eq(b, [1, 2, 3]) # b is unchanged
a = [1, 2, 3]
b = a
a += [4] # updates a (and thus b) in place
assert.eq(a, [1, 2, 3, 4])
assert.eq(b, [1, 2, 3, 4]) # alias observes the change
a = [1, 2, 3]
b = a
a.extend([4]) # updates existing list
assert.eq(a, [1, 2, 3, 4])
assert.eq(b, [1, 2, 3, 4]) # alias observes the change
list_extend()
# Unlike list.extend(iterable), list += iterable makes its LHS name local.
a_list = []
def f4():
a_list += [1] # binding use => a_list is a local var
assert.fails(f4, "local variable a_list referenced before assignment")
# list += <not iterable>
def f5():
x = []
x += 1
assert.fails(f5, "unknown binary op: list \\+ int")
# frozen list += iterable
def f6():
x = []
freeze(x)
x += [1]
assert.fails(f6, "cannot apply \\+= to frozen list")
# list += hasfields (hasfields is not iterable but defines list+hasfields)
def f7():
x = []
x += hasfields()
return x
assert.eq(f7(), 42) # weird, but exercises a corner case in list+=x.
# append
x5 = [1, 2, 3]
x5.append(4)
x5.append("abc")
assert.eq(x5, [1, 2, 3, 4, "abc"])
# extend
x5a = [1, 2, 3]
x5a.extend("abc".elems()) # string
x5a.extend((True, False)) # tuple
assert.eq(x5a, [1, 2, 3, "a", "b", "c", True, False])
# list.insert
def insert_at(index):
x = list(range(3))
x.insert(index, 42)
return x
assert.eq(insert_at(-99), [42, 0, 1, 2])
assert.eq(insert_at(-2), [0, 42, 1, 2])
assert.eq(insert_at(-1), [0, 1, 42, 2])
assert.eq(insert_at(0), [42, 0, 1, 2])
assert.eq(insert_at(1), [0, 42, 1, 2])
assert.eq(insert_at(2), [0, 1, 42, 2])
assert.eq(insert_at(3), [0, 1, 2, 42])
assert.eq(insert_at(4), [0, 1, 2, 42])
# list.remove
def remove(v):
x = [3, 1, 4, 1]
x.remove(v)
return x
assert.eq(remove(3), [1, 4, 1])
assert.eq(remove(1), [3, 4, 1])
assert.eq(remove(4), [3, 1, 1])
assert.fails(lambda: [3, 1, 4, 1].remove(42), "remove: element not found")
# list.index
bananas = list("bananas".elems())
assert.eq(bananas.index("a"), 1) # bAnanas
assert.fails(lambda: bananas.index("d"), "value not in list")
# start
assert.eq(bananas.index("a", -1000), 1) # bAnanas
assert.eq(bananas.index("a", 0), 1) # bAnanas
assert.eq(bananas.index("a", 1), 1) # bAnanas
assert.eq(bananas.index("a", 2), 3) # banAnas
assert.eq(bananas.index("a", 3), 3) # banAnas
assert.eq(bananas.index("b", 0), 0) # Bananas
assert.eq(bananas.index("n", -3), 4) # banaNas
assert.fails(lambda: bananas.index("n", -2), "value not in list")
assert.eq(bananas.index("s", -2), 6) # bananaS
assert.fails(lambda: bananas.index("b", 1), "value not in list")
# start, end
assert.eq(bananas.index("s", -1000, 7), 6) # bananaS
assert.fails(lambda: bananas.index("s", -1000, 6), "value not in list")
assert.fails(lambda: bananas.index("d", -1000, 1000), "value not in list")
# slicing, x[i:j:k]
assert.eq(bananas[6::-2], list("snnb".elems()))
assert.eq(bananas[5::-2], list("aaa".elems()))
assert.eq(bananas[4::-2], list("nnb".elems()))
assert.eq(bananas[99::-2], list("snnb".elems()))
assert.eq(bananas[100::-2], list("snnb".elems()))
# TODO(adonovan): many more tests
# iterator invalidation
def iterator1():
list = [0, 1, 2]
for x in list:
list[x] = 2 * x
return list
assert.fails(iterator1, "assign to element.* during iteration")
def iterator2():
list = [0, 1, 2]
for x in list:
list.remove(x)
assert.fails(iterator2, "remove.*during iteration")
def iterator3():
list = [0, 1, 2]
for x in list:
list.append(3)
assert.fails(iterator3, "append.*during iteration")
def iterator4():
list = [0, 1, 2]
for x in list:
list.extend([3, 4])
assert.fails(iterator4, "extend.*during iteration")
def iterator5():
def f(x):
x.append(4)
list = [1, 2, 3]
_ = [f(list) for x in list]
assert.fails(iterator5, "append.*during iteration")
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/math.star
Vendored
+379
View File
@@ -0,0 +1,379 @@
# Tests of math module.
load('math.star', 'math')
load('assert.star', 'assert')
def near(got, want, threshold):
return math.fabs(got-want) < threshold
inf, nan = float("inf"), float("nan")
# ceil
assert.eq(math.ceil(0.0), 0.0)
assert.eq(math.ceil(0.4), 1.0)
assert.eq(math.ceil(0.5), 1.0)
assert.eq(math.ceil(1.0), 1.0)
assert.eq(math.ceil(10.0), 10.0)
assert.eq(math.ceil(0), 0.0)
assert.eq(math.ceil(1), 1.0)
assert.eq(math.ceil(10), 10.0)
assert.eq(math.ceil(-0.0), 0.0)
assert.eq(math.ceil(-0.4), 0.0)
assert.eq(math.ceil(-0.5), 0.0)
assert.eq(math.ceil(-1.0), -1.0)
assert.eq(math.ceil(-10.0), -10.0)
assert.eq(math.ceil(-1), -1.0)
assert.eq(math.ceil(-10), -10.0)
assert.eq(type(math.ceil(0)), "int")
assert.eq(type(math.ceil(0.4)), "int")
assert.eq(type(math.ceil(10)), "int")
assert.eq(type(math.ceil(-10.0)), "int")
assert.eq(type(math.ceil(-0.5)), "int")
assert.eq(math.ceil((1<<63) + 0.5), int(float((1<<63) + 1)))
assert.fails(
lambda: math.ceil(inf), "cannot convert float infinity to integer")
assert.fails(
lambda: math.ceil(-inf), "cannot convert float infinity to integer")
assert.fails(
lambda: math.ceil(nan), "cannot convert float NaN to integer")
assert.fails(lambda: math.ceil("0"), "got string, want float or int")
# fabs
assert.eq(math.fabs(2.0), 2.0)
assert.eq(math.fabs(0.0), 0.0)
assert.eq(math.fabs(-2.0), 2.0)
assert.eq(math.fabs(2), 2)
assert.eq(math.fabs(0), 0)
assert.eq(math.fabs(-2), 2)
assert.eq(math.fabs(inf), inf)
assert.eq(math.fabs(-inf), inf)
assert.eq(math.fabs(nan), nan)
assert.fails(lambda: math.fabs("0"), "got string, want float or int")
# floor
assert.eq(math.floor(0.0), 0.0)
assert.eq(math.floor(0.4), 0.0)
assert.eq(math.floor(0.5), 0.0)
assert.eq(math.floor(1.0), 1.0)
assert.eq(math.floor(10.0), 10.0)
assert.eq(math.floor(-0.0), 0.0)
assert.eq(math.floor(-0.4), -1.0)
assert.eq(math.floor(-0.5), -1.0)
assert.eq(math.floor(-1.0), -1.0)
assert.eq(math.floor(-10.0), -10.0)
assert.eq(type(math.floor(0)), "int")
assert.eq(type(math.floor(0.4)), "int")
assert.eq(type(math.floor(10)), "int")
assert.eq(type(math.floor(-10.0)), "int")
assert.eq(type(math.floor(-0.5)), "int")
assert.eq(math.floor((1<<63) + 0.5), int(float(1<<63)))
assert.fails(
lambda: math.floor(inf), "cannot convert float infinity to integer")
assert.fails(
lambda: math.floor(-inf), "cannot convert float infinity to integer")
assert.fails(
lambda: math.floor(nan), "cannot convert float NaN to integer")
assert.fails(lambda: math.floor("0"), "got string, want float or int")
# mod
assert.eq(math.mod(5, 3), 2)
assert.eq(math.mod(inf, 1), nan)
assert.eq(math.mod(-inf, 1.0), nan)
assert.eq(math.mod(nan, 1.0), nan)
assert.eq(math.mod(1.0, 0.0), nan)
assert.eq(math.mod(1.0, inf), 1)
assert.eq(math.mod(1.0, -inf), 1)
assert.eq(math.mod(1.0, nan), nan)
assert.fails(lambda: math.mod("0", 1.0), "got string, want float or int")
assert.fails(lambda: math.mod(1.0, "0"), "got string, want float or int")
# pow
assert.eq(math.pow(5, 3), 125)
assert.eq(math.pow(5, 0), 1)
assert.eq(math.pow(5, 1), 5)
assert.eq(math.pow(1, 5), 1)
assert.eq(math.pow(inf, 1), inf)
assert.eq(math.pow(-inf, 1.0), -inf)
assert.eq(math.pow(nan, 1.0), nan)
assert.eq(math.pow(1.1, inf), inf)
assert.eq(math.pow(1.1, -inf), 0)
assert.eq(math.pow(2.0, nan), nan)
assert.fails(lambda: math.pow("0", 1.0), "got string, want float or int")
assert.fails(lambda: math.pow(1.0, "0"), "got string, want float or int")
# copysign
assert.eq(math.copysign(3.2, -1), -3.2)
assert.eq(math.copysign(inf, -1.0),-inf)
assert.eq(math.copysign(-inf, -1), -inf)
assert.eq(math.copysign(nan, -1), nan)
assert.eq(math.copysign(-1, nan), 1)
assert.fails(lambda: math.copysign("0", 1.0), "got string, want float or int")
assert.fails(lambda: math.copysign(1.0, "0"), "got string, want float or int")
# remainder
assert.eq(math.remainder(3, 5), -2)
assert.eq(math.remainder(1, 0), nan)
assert.eq(math.remainder(2, inf), 2)
assert.eq(math.remainder(2, -inf), 2)
assert.eq(math.remainder(inf, -1.0), nan)
assert.eq(math.remainder(-inf, -1), nan)
assert.eq(math.remainder(nan, -1), nan)
assert.eq(math.remainder(-1, nan), nan)
assert.fails(lambda: math.remainder("0", 1.0), "got string, want float or int")
assert.fails(lambda: math.remainder(1.0, "0"), "got string, want float or int")
# round
assert.eq(math.round(0.0), 0.0)
assert.eq(math.round(0.4), 0.0)
assert.eq(math.round(0.5), 1.0)
assert.eq(math.round(0.6), 1.0)
assert.eq(math.round(1.0), 1.0)
assert.eq(math.round(10.0), 10.0)
assert.eq(math.round(inf), inf)
assert.eq(math.round(nan), nan)
assert.eq(math.round(-0.4), 0.0)
assert.eq(math.round(-0.5), -1.0)
assert.eq(math.round(-0.6), -1.0)
assert.eq(math.round(-1.0), -1.0)
assert.eq(math.round(-10.0), -10.0)
assert.eq(math.round(-inf), -inf)
assert.fails(lambda: math.round("0"), "got string, want float or int")
# exp
assert.eq(math.exp(0.0), 1)
assert.eq(math.exp(1.0), math.e)
assert.true(near(math.exp(2.0), math.e * math.e, 0.00000000000001))
assert.eq(math.exp(-1.0), 1 / math.e)
assert.eq(math.exp(0), 1)
assert.eq(math.exp(1), math.e)
assert.true(near(math.exp(2), math.e * math.e, 0.00000000000001))
assert.eq(math.exp(-1), 1 / math.e)
assert.eq(math.exp(inf), inf)
assert.eq(math.exp(-inf), 0)
assert.eq(math.exp(nan), nan)
assert.fails(lambda: math.exp("0"), "got string, want float or int")
# sqrt
assert.eq(math.sqrt(0.0), 0.0)
assert.eq(math.sqrt(4.0), 2.0)
assert.eq(math.sqrt(-4.0), nan)
assert.eq(math.sqrt(0), 0)
assert.eq(math.sqrt(4), 2)
assert.eq(math.sqrt(-4), nan)
assert.eq(math.sqrt(nan), nan)
assert.eq(math.sqrt(inf), inf)
assert.eq(math.sqrt(-inf), nan)
assert.fails(lambda: math.sqrt("0"), "got string, want float or int")
# acos
assert.eq(math.acos(1.0), 0)
assert.eq(math.acos(1), 0)
assert.eq(math.acos(0.0), math.pi / 2)
assert.eq(math.acos(0), math.pi / 2)
assert.eq(math.acos(-1.0), math.pi)
assert.eq(math.acos(-1), math.pi)
assert.eq(math.acos(1.01), nan)
assert.eq(math.acos(-1.01), nan)
assert.eq(math.acos(inf), nan)
assert.eq(math.acos(-inf), nan)
assert.eq(math.acos(nan), nan)
assert.fails(lambda: math.acos("0"), "got string, want float or int")
# asin
assert.eq(math.asin(0.0), 0)
assert.eq(math.asin(1.0), math.pi / 2)
assert.eq(math.asin(-1.0), -math.pi / 2)
assert.eq(math.asin(0), 0)
assert.eq(math.asin(1), math.pi / 2)
assert.eq(math.asin(-1), -math.pi / 2)
assert.eq(math.asin(1.01), nan)
assert.eq(math.asin(-1.01), nan)
assert.eq(math.asin(inf), nan)
assert.eq(math.asin(-inf), nan)
assert.eq(math.asin(nan), nan)
assert.fails(lambda: math.asin("0"), "got string, want float or int")
# atan
assert.eq(math.atan(0.0), 0)
assert.eq(math.atan(1.0), math.pi / 4)
assert.eq(math.atan(-1.0), -math.pi / 4)
assert.eq(math.atan(1), math.pi / 4)
assert.eq(math.atan(-1), -math.pi / 4)
assert.eq(math.atan(inf), math.pi / 2)
assert.eq(math.atan(-inf), -math.pi / 2)
assert.eq(math.atan(nan), nan)
assert.fails(lambda: math.atan("0"), "got string, want float or int")
# atan2
assert.eq(math.atan2(1.0, 1.0), math.pi / 4)
assert.eq(math.atan2(-1.0, 1.0), -math.pi / 4)
assert.eq(math.atan2(0.0, 10.0), 0)
assert.eq(math.atan2(0.0, -10.0), math.pi)
assert.eq(math.atan2(-0.0, -10.0), -math.pi)
assert.eq(math.atan2(10.0, 0.0), math.pi / 2)
assert.eq(math.atan2(-10.0, 0.0), -math.pi / 2)
assert.eq(math.atan2(1, 1), math.pi / 4)
assert.eq(math.atan2(-1, 1), -math.pi / 4)
assert.eq(math.atan2(0, 10.0), 0)
assert.eq(math.atan2(0.0, -10), math.pi)
assert.eq(math.atan2(-0.0, -10), -math.pi)
assert.eq(math.atan2(10.0, 0), math.pi / 2)
assert.eq(math.atan2(-10.0, 0), -math.pi / 2)
assert.eq(math.atan2(1.0, nan), nan)
assert.eq(math.atan2(nan, 1.0), nan)
assert.eq(math.atan2(10.0, inf), 0)
assert.eq(math.atan2(-10.0, inf), 0)
assert.eq(math.atan2(10.0, -inf), math.pi)
assert.eq(math.atan2(-10.0, -inf), -math.pi)
assert.eq(math.atan2(inf, 10.0), math.pi / 2)
assert.eq(math.atan2(inf, -10.0), math.pi / 2)
assert.eq(math.atan2(-inf, 10.0), -math.pi / 2)
assert.eq(math.atan2(-inf, -10.0), -math.pi / 2)
assert.eq(math.atan2(inf, inf), math.pi / 4)
assert.eq(math.atan2(-inf, inf), -math.pi / 4)
assert.eq(math.atan2(inf, -inf), 3 * math.pi / 4)
assert.eq(math.atan2(-inf, -inf), -3 * math.pi / 4)
assert.fails(lambda: math.atan2("0", 1.0), "got string, want float or int")
assert.fails(lambda: math.atan2(1.0, "0"), "got string, want float or int")
# cos
assert.eq(math.cos(0.0), 1)
assert.true(near(math.cos(math.pi / 2), 0, 0.00000000000001))
assert.eq(math.cos(math.pi), -1)
assert.true(near(math.cos(-math.pi / 2), 0, 0.00000000000001))
assert.eq(math.cos(-math.pi), -1)
assert.eq(math.cos(inf), nan)
assert.eq(math.cos(-inf), nan)
assert.eq(math.cos(nan), nan)
assert.fails(lambda: math.cos("0"), "got string, want float or int")
# hypot
assert.eq(math.hypot(4.0, 3.0), 5.0)
assert.eq(math.hypot(4, 3), 5.0)
assert.eq(math.hypot(inf, 3.0), inf)
assert.eq(math.hypot(-inf, 3.0), inf)
assert.eq(math.hypot(3.0, inf), inf)
assert.eq(math.hypot(3.0, -inf), inf)
assert.eq(math.hypot(nan, 3.0), nan)
assert.eq(math.hypot(3.0, nan), nan)
assert.fails(lambda: math.hypot("0", 1.0), "got string, want float or int")
assert.fails(lambda: math.hypot(1.0, "0"), "got string, want float or int")
# sin
assert.eq(math.sin(0.0), 0)
assert.eq(math.sin(0), 0)
assert.eq(math.sin(math.pi / 2), 1)
assert.eq(math.sin(-math.pi / 2), -1)
assert.eq(math.sin(inf), nan)
assert.eq(math.sin(-inf), nan)
assert.eq(math.sin(nan), nan)
assert.fails(lambda: math.sin("0"), "got string, want float or int")
# tan
assert.eq(math.tan(0.0), 0)
assert.eq(math.tan(0), 0)
assert.true(near(math.tan(math.pi / 4), 1, 0.00000000000001))
assert.true(near(math.tan(-math.pi / 4), -1, 0.00000000000001))
assert.eq(math.tan(inf), nan)
assert.eq(math.tan(-inf), nan)
assert.eq(math.tan(nan), nan)
assert.fails(lambda: math.tan("0"), "got string, want float or int")
# degrees
oneDeg = 57.29577951308232
assert.eq(math.degrees(1.0), oneDeg)
assert.eq(math.degrees(1), oneDeg)
assert.eq(math.degrees(-1.0), -oneDeg)
assert.eq(math.degrees(-1), -oneDeg)
assert.eq(math.degrees(inf), inf)
assert.eq(math.degrees(-inf), -inf)
assert.eq(math.degrees(nan), nan)
assert.fails(lambda: math.degrees("0"), "got string, want float or int")
# radians
oneRad = 0.017453292519943295
assert.eq(math.radians(1.0), oneRad)
assert.eq(math.radians(-1.0), -oneRad)
assert.eq(math.radians(1), oneRad)
assert.eq(math.radians(-1), -oneRad)
assert.eq(math.radians(inf), inf)
assert.eq(math.radians(-inf), -inf)
assert.eq(math.radians(nan), nan)
assert.fails(lambda: math.radians("0"), "got string, want float or int")
# acosh
assert.eq(math.acosh(1.0), 0)
assert.eq(math.acosh(1), 0)
assert.eq(math.acosh(0.99), nan)
assert.eq(math.acosh(0), nan)
assert.eq(math.acosh(-0.99), nan)
assert.eq(math.acosh(-inf), nan)
assert.eq(math.acosh(inf), inf)
assert.eq(math.acosh(nan), nan)
assert.fails(lambda: math.acosh("0"), "got string, want float or int")
# asinh
asinhOne = 0.8813735870195432
assert.eq(math.asinh(0.0), 0)
assert.eq(math.asinh(0), 0)
assert.true(near(math.asinh(1.0), asinhOne, 0.00000001))
assert.true(near(math.asinh(1), asinhOne, 0.00000001))
assert.true(near(math.asinh(-1.0), -asinhOne, 0.00000001))
assert.true(near(math.asinh(-1), -asinhOne, 0.00000001))
assert.eq(math.asinh(inf), inf)
assert.eq(math.asinh(-inf), -inf)
assert.eq(math.asinh(nan), nan)
assert.fails(lambda: math.asinh("0"), "got string, want float or int")
# atanh
atanhHalf = 0.5493061443340548
assert.eq(math.atanh(0.0), 0)
assert.eq(math.atanh(0), 0)
assert.eq(math.atanh(0.5), atanhHalf)
assert.eq(math.atanh(-0.5), -atanhHalf)
assert.eq(math.atanh(1), inf)
assert.eq(math.atanh(-1), -inf)
assert.eq(math.atanh(1.1), nan)
assert.eq(math.atanh(-1.1), nan)
assert.eq(math.atanh(inf), nan)
assert.eq(math.atanh(-inf), nan)
assert.eq(math.atanh(nan), nan)
assert.fails(lambda: math.atanh("0"), "got string, want float or int")
# cosh
coshOne = 1.5430806348152437
assert.eq(math.cosh(1.0), coshOne)
assert.eq(math.cosh(1), coshOne)
assert.eq(math.cosh(0.0), 1)
assert.eq(math.cosh(0), 1)
assert.eq(math.cosh(-inf), inf)
assert.eq(math.cosh(inf), inf)
assert.eq(math.cosh(nan), nan)
assert.fails(lambda: math.cosh("0"), "got string, want float or int")
# sinh
sinhOne = 1.1752011936438014
assert.eq(math.sinh(0.0), 0)
assert.eq(math.sinh(0), 0)
assert.eq(math.sinh(1.0), sinhOne)
assert.eq(math.sinh(1), sinhOne)
assert.eq(math.sinh(-1.0), -sinhOne)
assert.eq(math.sinh(-1), -sinhOne)
assert.eq(math.sinh(-inf), -inf)
assert.eq(math.sinh(inf), inf)
assert.eq(math.sinh(nan), nan)
assert.fails(lambda: math.sinh("0"), "got string, want float or int")
# tanh
tanhOne = 0.7615941559557649
assert.eq(math.tanh(0.0), 0)
assert.eq(math.tanh(0), 0)
assert.eq(math.tanh(1.0), tanhOne)
assert.eq(math.tanh(1), tanhOne)
assert.eq(math.tanh(-1.0), -tanhOne)
assert.eq(math.tanh(-1), -tanhOne)
assert.eq(math.tanh(-inf), -1)
assert.eq(math.tanh(inf), 1)
assert.eq(math.tanh(nan), nan)
assert.fails(lambda: math.tanh("0"), "got string, want float or int")
# log
assert.eq(math.log(math.e), 1)
assert.eq(math.log(10, 10), 1)
assert.eq(math.log(10.0, 10.0), 1)
assert.eq(math.log(2, 2.0), 1)
assert.fails(lambda: math.log(2, 1), "division by zero")
assert.fails(lambda: math.log(0.99, 1.0), "division by zero")
assert.eq(math.log(0.0), -inf)
assert.eq(math.log(0), -inf)
assert.eq(math.log(-1.0), nan)
assert.eq(math.log(-1), nan)
assert.eq(math.log(nan), nan)
assert.fails(lambda: math.log("0"), "got string, want float or int")
assert.fails(lambda: math.log(10, "10"), "got string, want float or int")
# gamma
assert.eq(math.gamma(1.0), 1)
assert.eq(math.gamma(1), 1)
assert.eq(math.gamma(-1), nan)
assert.eq(math.gamma(0), inf)
assert.eq(math.gamma(-inf), nan)
assert.eq(math.gamma(inf), inf)
assert.eq(math.gamma(nan), nan)
assert.fails(lambda: math.gamma("0"), "got string, want float or int")
# Constants
assert.eq(math.e, 2.7182818284590452)
assert.eq(math.pi, 3.1415926535897932)
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/misc.star
Vendored
+139
View File
@@ -0,0 +1,139 @@
# Miscellaneous tests of Starlark evaluation.
# This is a "chunked" file: each "---" effectively starts a new file.
# TODO(adonovan): move these tests into more appropriate files.
# TODO(adonovan): test coverage:
# - stmts: pass; if cond fail; += and failures;
# for x fail; for x not iterable; for can't assign; for
# error in loop body
# - subassign fail
# - x[i]=x fail in both operands; frozen x; list index not int; boundscheck
# - x.f = ...
# - failure in list expr [...]; tuple expr; dict expr (bad key)
# - cond expr semantics; failures
# - x[i] failures in both args; dict and iterator key and range checks;
# unhandled operand types
# - +: list/list, int/int, string/string, tuple+tuple, dict/dict;
# - * and ** calls: various errors
# - call of non-function
# - slice x[ijk]
# - comprehension: unhashable dict key;
# scope of vars (local and toplevel); noniterable for clause
# - unknown unary op
# - ordering of values
# - freeze, transitivity of its effect.
# - add an application-defined type to the environment so we can test it.
# - even more:
#
# eval
# pass statement
# assign to tuple l-value -- illegal
# assign to list l-value -- illegal
# assign to field
# tuple + tuple
# call with *args, **kwargs
# slice with step
# tuple slice
# interpolate with %c, %%
load("assert.star", "assert")
# Ordered comparisons require values of the same type.
assert.fails(lambda: None < None, "not impl")
assert.fails(lambda: None < False, "not impl")
assert.fails(lambda: False < list, "not impl")
assert.fails(lambda: list < {}, "not impl")
assert.fails(lambda: {} < (lambda: None), "not impl")
assert.fails(lambda: (lambda: None) < 0, "not impl")
assert.fails(lambda: 0 < [], "not impl")
assert.fails(lambda: [] < "", "not impl")
assert.fails(lambda: "" < (), "not impl")
# Except int < float:
assert.lt(1, 2.0)
assert.lt(2.0, 3)
---
# cyclic data structures
load("assert.star", "assert")
cyclic = [1, 2, 3] # list cycle
cyclic[1] = cyclic
assert.eq(str(cyclic), "[1, [...], 3]")
assert.fails(lambda: cyclic < cyclic, "maximum recursion")
assert.fails(lambda: cyclic == cyclic, "maximum recursion")
cyclic2 = [1, 2, 3]
cyclic2[1] = cyclic2
assert.fails(lambda: cyclic2 == cyclic, "maximum recursion")
cyclic3 = [1, [2, 3]] # list-list cycle
cyclic3[1][0] = cyclic3
assert.eq(str(cyclic3), "[1, [[...], 3]]")
cyclic4 = {"x": 1}
cyclic4["x"] = cyclic4
assert.eq(str(cyclic4), "{\"x\": {...}}")
cyclic5 = [0, {"x": 1}] # list-dict cycle
cyclic5[1]["x"] = cyclic5
assert.eq(str(cyclic5), "[0, {\"x\": [...]}]")
assert.eq(str(cyclic5), "[0, {\"x\": [...]}]")
assert.fails(lambda: cyclic5 == cyclic5 ,"maximum recursion")
cyclic6 = [0, {"x": 1}]
cyclic6[1]["x"] = cyclic6
assert.fails(lambda: cyclic5 == cyclic6, "maximum recursion")
---
# regression
load("assert.star", "assert")
# was a parse error:
assert.eq(("ababab"[2:]).replace("b", "c"), "acac")
assert.eq("ababab"[2:].replace("b", "c"), "acac")
# test parsing of line continuation, at toplevel and in expression.
three = 1 + \
2
assert.eq(1 + \
2, three)
---
# A regression test for error position information.
_ = {}.get(1, default=2) ### "get: unexpected keyword arguments"
---
# Load exposes explicitly declared globals from other modules.
load('assert.star', 'assert', 'freeze')
assert.eq(str(freeze), '<built-in function freeze>')
---
# Load does not expose pre-declared globals from other modules.
# See github.com/google/skylark/issues/75.
load('assert.star', 'assert', 'matches') ### "matches not found in module"
---
# Load does not expose universals accessible in other modules.
load('assert.star', 'len') ### "len not found in module"
---
# Test plus folding optimization.
load('assert.star', 'assert')
s = "s"
l = [4]
t = (4,)
assert.eq("a" + "b" + "c", "abc")
assert.eq("a" + "b" + s + "c", "absc")
assert.eq(() + (1,) + (2, 3), (1, 2, 3))
assert.eq(() + (1,) + t + (2, 3), (1, 4, 2, 3))
assert.eq([] + [1] + [2, 3], [1, 2, 3])
assert.eq([] + [1] + l + [2, 3], [1, 4, 2, 3])
assert.fails(lambda: "a" + "b" + 1 + "c", "unknown binary op: string \\+ int")
assert.fails(lambda: () + () + 1 + (), "unknown binary op: tuple \\+ int")
assert.fails(lambda: [] + [] + 1 + [], "unknown binary op: list \\+ int")
---
load('assert.star', 'froze') ### `name froze not found .*did you mean freeze`
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/module.star
Vendored
+17
View File
@@ -0,0 +1,17 @@
# Tests of Module.
load("assert.star", "assert")
assert.eq(type(assert), "module")
assert.eq(str(assert), '<module "assert">')
assert.eq(dir(assert), ["contains", "eq", "fail", "fails", "lt", "ne", "true"])
assert.fails(lambda : {assert: None}, "unhashable: module")
def assignfield():
assert.foo = None
assert.fails(assignfield, "can't assign to .foo field of module")
# no such field
assert.fails(lambda : assert.nonesuch, "module has no .nonesuch field or method$")
assert.fails(lambda : assert.falls, "module has no .falls field or method .did you mean .fails\\?")
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/paths.star
Vendored
+250
View File
@@ -0,0 +1,250 @@
# Copyright 2017 The Bazel Authors. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Skylib module containing file path manipulation functions.
NOTE: The functions in this module currently only support paths with Unix-style
path separators (forward slash, "/"); they do not handle Windows-style paths
with backslash separators or drive letters.
"""
# This file is in the Bazel build language dialect of Starlark,
# so declarations of 'fail' and 'struct' are required to make
# it compile in the core language.
def fail(msg):
print(msg)
struct = dict
def _basename(p):
"""Returns the basename (i.e., the file portion) of a path.
Note that if `p` ends with a slash, this function returns an empty string.
This matches the behavior of Python's `os.path.basename`, but differs from
the Unix `basename` command (which would return the path segment preceding
the final slash).
Args:
p: The path whose basename should be returned.
Returns:
The basename of the path, which includes the extension.
"""
return p.rpartition("/")[-1]
def _dirname(p):
"""Returns the dirname of a path.
The dirname is the portion of `p` up to but not including the file portion
(i.e., the basename). Any slashes immediately preceding the basename are not
included, unless omitting them would make the dirname empty.
Args:
p: The path whose dirname should be returned.
Returns:
The dirname of the path.
"""
prefix, sep, _ = p.rpartition("/")
if not prefix:
return sep
else:
# If there are multiple consecutive slashes, strip them all out as Python's
# os.path.dirname does.
return prefix.rstrip("/")
def _is_absolute(path):
"""Returns `True` if `path` is an absolute path.
Args:
path: A path (which is a string).
Returns:
`True` if `path` is an absolute path.
"""
return path.startswith("/") or (len(path) > 2 and path[1] == ":")
def _join(path, *others):
"""Joins one or more path components intelligently.
This function mimics the behavior of Python's `os.path.join` function on POSIX
platform. It returns the concatenation of `path` and any members of `others`,
inserting directory separators before each component except the first. The
separator is not inserted if the path up until that point is either empty or
already ends in a separator.
If any component is an absolute path, all previous components are discarded.
Args:
path: A path segment.
*others: Additional path segments.
Returns:
A string containing the joined paths.
"""
result = path
for p in others:
if _is_absolute(p):
result = p
elif not result or result.endswith("/"):
result += p
else:
result += "/" + p
return result
def _normalize(path):
"""Normalizes a path, eliminating double slashes and other redundant segments.
This function mimics the behavior of Python's `os.path.normpath` function on
POSIX platforms; specifically:
- If the entire path is empty, "." is returned.
- All "." segments are removed, unless the path consists solely of a single
"." segment.
- Trailing slashes are removed, unless the path consists solely of slashes.
- ".." segments are removed as long as there are corresponding segments
earlier in the path to remove; otherwise, they are retained as leading ".."
segments.
- Single and double leading slashes are preserved, but three or more leading
slashes are collapsed into a single leading slash.
- Multiple adjacent internal slashes are collapsed into a single slash.
Args:
path: A path.
Returns:
The normalized path.
"""
if not path:
return "."
if path.startswith("//") and not path.startswith("///"):
initial_slashes = 2
elif path.startswith("/"):
initial_slashes = 1
else:
initial_slashes = 0
is_relative = (initial_slashes == 0)
components = path.split("/")
new_components = []
for component in components:
if component in ("", "."):
continue
if component == "..":
if new_components and new_components[-1] != "..":
# Only pop the last segment if it isn't another "..".
new_components.pop()
elif is_relative:
# Preserve leading ".." segments for relative paths.
new_components.append(component)
else:
new_components.append(component)
path = "/".join(new_components)
if not is_relative:
path = ("/" * initial_slashes) + path
return path or "."
def _relativize(path, start):
"""Returns the portion of `path` that is relative to `start`.
Because we do not have access to the underlying file system, this
implementation differs slightly from Python's `os.path.relpath` in that it
will fail if `path` is not beneath `start` (rather than use parent segments to
walk up to the common file system root).
Relativizing paths that start with parent directory references only works if
the path both start with the same initial parent references.
Args:
path: The path to relativize.
start: The ancestor path against which to relativize.
Returns:
The portion of `path` that is relative to `start`.
"""
segments = _normalize(path).split("/")
start_segments = _normalize(start).split("/")
if start_segments == ["."]:
start_segments = []
start_length = len(start_segments)
if (path.startswith("/") != start.startswith("/") or
len(segments) < start_length):
fail("Path '%s' is not beneath '%s'" % (path, start))
for ancestor_segment, segment in zip(start_segments, segments):
if ancestor_segment != segment:
fail("Path '%s' is not beneath '%s'" % (path, start))
length = len(segments) - start_length
result_segments = segments[-length:]
return "/".join(result_segments)
def _replace_extension(p, new_extension):
"""Replaces the extension of the file at the end of a path.
If the path has no extension, the new extension is added to it.
Args:
p: The path whose extension should be replaced.
new_extension: The new extension for the file. The new extension should
begin with a dot if you want the new filename to have one.
Returns:
The path with the extension replaced (or added, if it did not have one).
"""
return _split_extension(p)[0] + new_extension
def _split_extension(p):
"""Splits the path `p` into a tuple containing the root and extension.
Leading periods on the basename are ignored, so
`path.split_extension(".bashrc")` returns `(".bashrc", "")`.
Args:
p: The path whose root and extension should be split.
Returns:
A tuple `(root, ext)` such that the root is the path without the file
extension, and `ext` is the file extension (which, if non-empty, contains
the leading dot). The returned tuple always satisfies the relationship
`root + ext == p`.
"""
b = _basename(p)
last_dot_in_basename = b.rfind(".")
# If there is no dot or the only dot in the basename is at the front, then
# there is no extension.
if last_dot_in_basename <= 0:
return (p, "")
dot_distance_from_end = len(b) - last_dot_in_basename
return (p[:-dot_distance_from_end], p[-dot_distance_from_end:])
paths = struct(
basename = _basename,
dirname = _dirname,
is_absolute = _is_absolute,
join = _join,
normalize = _normalize,
relativize = _relativize,
replace_extension = _replace_extension,
split_extension = _split_extension,
)
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/proto.star
Vendored
+14
View File
@@ -0,0 +1,14 @@
# Tests of the experimental 'lib/proto' module.
load("assert.star", "assert")
load("proto.star", "proto")
schema = proto.file("google/protobuf/descriptor.proto")
m = schema.FileDescriptorProto(name = "somename.proto", dependency = ["a", "b", "c"])
assert.eq(type(m), "proto.Message")
assert.eq(m.name, "somename.proto")
assert.eq(list(m.dependency), ["a", "b", "c"])
m.dependency = ["d", "e"]
assert.eq(list(m.dependency), ["d", "e"])
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/recursion.star
Vendored
+14
View File
@@ -0,0 +1,14 @@
# Tests of Starlark recursion and while statement.
# This is a "chunked" file: each "---" effectively starts a new file.
# option:recursion
load("assert.star", "assert")
def fib(n):
if n <= 1:
return 1
return fib(n-1) + fib(n-2)
assert.eq(fib(5), 8)
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/set.star
Vendored
+198
View File
@@ -0,0 +1,198 @@
# Tests of Starlark 'set'
# option:set option:globalreassign
# Sets are not a standard part of Starlark, so the features
# tested in this file must be enabled in the application by setting
# resolve.AllowSet. (All sets are created by calls to the 'set'
# built-in or derived from operations on existing sets.)
# The semantics are subject to change as the spec evolves.
# TODO(adonovan): support set mutation:
# - del set[k]
# - set.update
# - set += iterable, perhaps?
# Test iterator invalidation.
load("assert.star", "assert", "freeze")
# literals
# Parser does not currently support {1, 2, 3}.
# TODO(adonovan): add test to syntax/testdata/errors.star.
# set comprehensions
# Parser does not currently support {x for x in y}.
# See syntax/testdata/errors.star.
# set constructor
assert.eq(type(set()), "set")
assert.eq(list(set()), [])
assert.eq(type(set([1, 3, 2, 3])), "set")
assert.eq(list(set([1, 3, 2, 3])), [1, 3, 2])
assert.eq(type(set("hello".elems())), "set")
assert.eq(list(set("hello".elems())), ["h", "e", "l", "o"])
assert.eq(list(set(range(3))), [0, 1, 2])
assert.fails(lambda : set(1), "got int, want iterable")
assert.fails(lambda : set(1, 2, 3), "got 3 arguments")
assert.fails(lambda : set([1, 2, {}]), "unhashable type: dict")
# truth
assert.true(not set())
assert.true(set([False]))
assert.true(set([1, 2, 3]))
x = set([1, 2, 3])
y = set([3, 4, 5])
# set + any is not defined
assert.fails(lambda : x + y, "unknown.*: set \\+ set")
# set | set
assert.eq(list(set("a".elems()) | set("b".elems())), ["a", "b"])
assert.eq(list(set("ab".elems()) | set("bc".elems())), ["a", "b", "c"])
assert.fails(lambda : set() | [], "unknown binary op: set | list")
assert.eq(type(x | y), "set")
assert.eq(list(x | y), [1, 2, 3, 4, 5])
assert.eq(list(x | set([5, 1])), [1, 2, 3, 5])
assert.eq(list(x | set((6, 5, 4))), [1, 2, 3, 6, 5, 4])
# set.union (allows any iterable for right operand)
assert.eq(list(set("a".elems()).union("b".elems())), ["a", "b"])
assert.eq(list(set("ab".elems()).union("bc".elems())), ["a", "b", "c"])
assert.eq(set().union([]), set())
assert.eq(type(x.union(y)), "set")
assert.eq(list(x.union(y)), [1, 2, 3, 4, 5])
assert.eq(list(x.union([5, 1])), [1, 2, 3, 5])
assert.eq(list(x.union((6, 5, 4))), [1, 2, 3, 6, 5, 4])
assert.fails(lambda : x.union([1, 2, {}]), "unhashable type: dict")
# intersection, set & set or set.intersection(iterable)
assert.eq(list(set("a".elems()) & set("b".elems())), [])
assert.eq(list(set("ab".elems()) & set("bc".elems())), ["b"])
assert.eq(list(set("a".elems()).intersection("b".elems())), [])
assert.eq(list(set("ab".elems()).intersection("bc".elems())), ["b"])
# symmetric difference, set ^ set or set.symmetric_difference(iterable)
assert.eq(set([1, 2, 3]) ^ set([4, 5, 3]), set([1, 2, 4, 5]))
assert.eq(set([1,2,3,4]).symmetric_difference([3,4,5,6]), set([1,2,5,6]))
assert.eq(set([1,2,3,4]).symmetric_difference(set([])), set([1,2,3,4]))
def test_set_augmented_assign():
x = set([1, 2, 3])
x &= set([2, 3])
assert.eq(x, set([2, 3]))
x |= set([1])
assert.eq(x, set([1, 2, 3]))
x ^= set([4, 5, 3])
assert.eq(x, set([1, 2, 4, 5]))
test_set_augmented_assign()
# len
assert.eq(len(x), 3)
assert.eq(len(y), 3)
assert.eq(len(x | y), 5)
# str
assert.eq(str(set([1])), "set([1])")
assert.eq(str(set([2, 3])), "set([2, 3])")
assert.eq(str(set([3, 2])), "set([3, 2])")
# comparison
assert.eq(x, x)
assert.eq(y, y)
assert.true(x != y)
assert.eq(set([1, 2, 3]), set([3, 2, 1]))
# iteration
assert.true(type([elem for elem in x]), "list")
assert.true(list([elem for elem in x]), [1, 2, 3])
def iter():
list = []
for elem in x:
list.append(elem)
return list
assert.eq(iter(), [1, 2, 3])
# sets are not indexable
assert.fails(lambda : x[0], "unhandled.*operation")
# adding and removing
add_set = set([1,2,3])
add_set.add(4)
assert.true(4 in add_set)
freeze(add_set) # no mutation of frozen set because key already present
add_set.add(4)
assert.fails(lambda: add_set.add(5), "add: cannot insert into frozen hash table")
# remove
remove_set = set([1,2,3])
remove_set.remove(3)
assert.true(3 not in remove_set)
assert.fails(lambda: remove_set.remove(3), "remove: missing key")
freeze(remove_set)
assert.fails(lambda: remove_set.remove(3), "remove: cannot delete from frozen hash table")
# discard
discard_set = set([1,2,3])
discard_set.discard(3)
assert.true(3 not in discard_set)
assert.eq(discard_set.discard(3), None)
freeze(discard_set)
assert.eq(discard_set.discard(3), None) # no mutation of frozen set because key doesn't exist
assert.fails(lambda: discard_set.discard(1), "discard: cannot delete from frozen hash table")
# pop
pop_set = set([1,2,3])
assert.eq(pop_set.pop(), 1)
assert.eq(pop_set.pop(), 2)
assert.eq(pop_set.pop(), 3)
assert.fails(lambda: pop_set.pop(), "pop: empty set")
pop_set.add(1)
pop_set.add(2)
freeze(pop_set)
assert.fails(lambda: pop_set.pop(), "pop: cannot delete from frozen hash table")
# clear
clear_set = set([1,2,3])
clear_set.clear()
assert.eq(len(clear_set), 0)
freeze(clear_set) # no mutation of frozen set because its already empty
assert.eq(clear_set.clear(), None)
other_clear_set = set([1,2,3])
freeze(other_clear_set)
assert.fails(lambda: other_clear_set.clear(), "clear: cannot clear frozen hash table")
# difference: set - set or set.difference(iterable)
assert.eq(set([1,2,3,4]).difference([1,2,3,4]), set([]))
assert.eq(set([1,2,3,4]).difference([1,2]), set([3,4]))
assert.eq(set([1,2,3,4]).difference([]), set([1,2,3,4]))
assert.eq(set([1,2,3,4]).difference(set([1,2,3])), set([4]))
assert.eq(set([1,2,3,4]) - set([1,2,3,4]), set())
assert.eq(set([1,2,3,4]) - set([1,2]), set([3,4]))
# issuperset: set >= set or set.issuperset(iterable)
assert.true(set([1,2,3]).issuperset([1,2]))
assert.true(not set([1,2,3]).issuperset(set([1,2,4])))
assert.true(set([1,2,3]) >= set([1,2,3]))
assert.true(set([1,2,3]) >= set([1,2]))
assert.true(not set([1,2,3]) >= set([1,2,4]))
# proper superset: set > set
assert.true(set([1, 2, 3]) > set([1, 2]))
assert.true(not set([1,2, 3]) > set([1, 2, 3]))
# issubset: set <= set or set.issubset(iterable)
assert.true(set([1,2]).issubset([1,2,3]))
assert.true(not set([1,2,3]).issubset(set([1,2,4])))
assert.true(set([1,2,3]) <= set([1,2,3]))
assert.true(set([1,2]) <= set([1,2,3]))
assert.true(not set([1,2,3]) <= set([1,2,4]))
# proper subset: set < set
assert.true(set([1,2]) < set([1,2,3]))
assert.true(not set([1,2,3]) < set([1,2,3]))
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/string.star
Vendored
+493
View File
@@ -0,0 +1,493 @@
# Tests of Starlark 'string'
# option:set
load("assert.star", "assert")
# raw string literals:
assert.eq(r"a\bc", "a\\bc")
# truth
assert.true("abc")
assert.true(chr(0))
assert.true(not "")
# str + str
assert.eq("a" + "b" + "c", "abc")
# str * int, int * str
assert.eq("abc" * 0, "")
assert.eq("abc" * -1, "")
assert.eq("abc" * 1, "abc")
assert.eq("abc" * 5, "abcabcabcabcabc")
assert.eq(0 * "abc", "")
assert.eq(-1 * "abc", "")
assert.eq(1 * "abc", "abc")
assert.eq(5 * "abc", "abcabcabcabcabc")
assert.fails(lambda: 1.0 * "abc", "unknown.*float \\* str")
assert.fails(lambda: "abc" * (1000000 * 1000000), "repeat count 1000000000000 too large")
assert.fails(lambda: "abc" * 1000000 * 1000000, "excessive repeat \\(3000000 \\* 1000000 elements")
# len
assert.eq(len("Hello, 世界!"), 14)
assert.eq(len("𐐷"), 4) # U+10437 has a 4-byte UTF-8 encoding (and a 2-code UTF-16 encoding)
# chr & ord
assert.eq(chr(65), "A") # 1-byte UTF-8 encoding
assert.eq(chr(1049), "Й") # 2-byte UTF-8 encoding
assert.eq(chr(0x1F63F), "😿") # 4-byte UTF-8 encoding
assert.fails(lambda: chr(-1), "Unicode code point -1 out of range \\(<0\\)")
assert.fails(lambda: chr(0x110000), "Unicode code point U\\+110000 out of range \\(>0x10FFFF\\)")
assert.eq(ord("A"), 0x41)
assert.eq(ord("Й"), 0x419)
assert.eq(ord("世"), 0x4e16)
assert.eq(ord("😿"), 0x1F63F)
assert.eq(ord("Й"[1:]), 0xFFFD) # = Unicode replacement character
assert.fails(lambda: ord("abc"), "string encodes 3 Unicode code points, want 1")
assert.fails(lambda: ord(""), "string encodes 0 Unicode code points, want 1")
assert.fails(lambda: ord("😿"[1:]), "string encodes 3 Unicode code points, want 1") # 3 x 0xFFFD
# string.codepoint_ords
assert.eq(type("abcЙ😿".codepoint_ords()), "string.codepoints")
assert.eq(str("abcЙ😿".codepoint_ords()), '"abcЙ😿".codepoint_ords()')
assert.eq(list("abcЙ😿".codepoint_ords()), [97, 98, 99, 1049, 128575])
assert.eq(list(("A" + "😿Z"[1:]).codepoint_ords()), [ord("A"), 0xFFFD, 0xFFFD, 0xFFFD, ord("Z")])
assert.eq(list("".codepoint_ords()), [])
assert.fails(lambda: "abcЙ😿".codepoint_ords()[2], "unhandled index") # not indexable
assert.fails(lambda: len("abcЙ😿".codepoint_ords()), "no len") # unknown length
# string.codepoints
assert.eq(type("abcЙ😿".codepoints()), "string.codepoints")
assert.eq(str("abcЙ😿".codepoints()), '"abcЙ😿".codepoints()')
assert.eq(list("abcЙ😿".codepoints()), ["a", "b", "c", "Й", "😿"])
assert.eq(list(("A" + "😿Z"[1:]).codepoints()), ["A", "�", "�", "�", "Z"])
assert.eq(list("".codepoints()), [])
assert.fails(lambda: "abcЙ😿".codepoints()[2], "unhandled index") # not indexable
assert.fails(lambda: len("abcЙ😿".codepoints()), "no len") # unknown length
# string.elem_ords
assert.eq(type("abcЙ😿".elem_ords()), "string.elems")
assert.eq(str("abcЙ😿".elem_ords()), '"abcЙ😿".elem_ords()')
assert.eq(list("abcЙ😿".elem_ords()), [97, 98, 99, 208, 153, 240, 159, 152, 191])
assert.eq(list(("A" + "😿Z"[1:]).elem_ords()), [65, 159, 152, 191, 90])
assert.eq(list("".elem_ords()), [])
assert.eq("abcЙ😿".elem_ords()[2], 99) # indexable
assert.eq(len("abcЙ😿".elem_ords()), 9) # known length
# string.elems (1-byte substrings, which are invalid text)
assert.eq(type("abcЙ😿".elems()), "string.elems")
assert.eq(str("abcЙ😿".elems()), '"abcЙ😿".elems()')
assert.eq(
repr(list("abcЙ😿".elems())),
r'["a", "b", "c", "\xd0", "\x99", "\xf0", "\x9f", "\x98", "\xbf"]',
)
assert.eq(
repr(list(("A" + "😿Z"[1:]).elems())),
r'["A", "\x9f", "\x98", "\xbf", "Z"]',
)
assert.eq(list("".elems()), [])
assert.eq("abcЙ😿".elems()[2], "c") # indexable
assert.eq(len("abcЙ😿".elems()), 9) # known length
# indexing, x[i]
assert.eq("Hello, 世界!"[0], "H")
assert.eq(repr("Hello, 世界!"[7]), r'"\xe4"') # (invalid text)
assert.eq("Hello, 世界!"[13], "!")
assert.fails(lambda: "abc"[-4], "out of range")
assert.eq("abc"[-3], "a")
assert.eq("abc"[-2], "b")
assert.eq("abc"[-1], "c")
assert.eq("abc"[0], "a")
assert.eq("abc"[1], "b")
assert.eq("abc"[2], "c")
assert.fails(lambda: "abc"[4], "out of range")
# x[i] = ...
def f():
"abc"[1] = "B"
assert.fails(f, "string.*does not support.*assignment")
# slicing, x[i:j]
assert.eq("abc"[:], "abc")
assert.eq("abc"[-4:], "abc")
assert.eq("abc"[-3:], "abc")
assert.eq("abc"[-2:], "bc")
assert.eq("abc"[-1:], "c")
assert.eq("abc"[0:], "abc")
assert.eq("abc"[1:], "bc")
assert.eq("abc"[2:], "c")
assert.eq("abc"[3:], "")
assert.eq("abc"[4:], "")
assert.eq("abc"[:-4], "")
assert.eq("abc"[:-3], "")
assert.eq("abc"[:-2], "a")
assert.eq("abc"[:-1], "ab")
assert.eq("abc"[:0], "")
assert.eq("abc"[:1], "a")
assert.eq("abc"[:2], "ab")
assert.eq("abc"[:3], "abc")
assert.eq("abc"[:4], "abc")
assert.eq("abc"[1:2], "b")
assert.eq("abc"[2:1], "")
assert.eq(repr("😿"[:1]), r'"\xf0"') # (invalid text)
# non-unit strides
assert.eq("abcd"[0:4:1], "abcd")
assert.eq("abcd"[::2], "ac")
assert.eq("abcd"[1::2], "bd")
assert.eq("abcd"[4:0:-1], "dcb")
assert.eq("banana"[7::-2], "aaa")
assert.eq("banana"[6::-2], "aaa")
assert.eq("banana"[5::-2], "aaa")
assert.eq("banana"[4::-2], "nnb")
assert.eq("banana"[::-1], "ananab")
assert.eq("banana"[None:None:-2], "aaa")
assert.fails(lambda: "banana"[1.0::], "invalid start index: got float, want int")
assert.fails(lambda: "banana"[:"":], "invalid end index: got string, want int")
assert.fails(lambda: "banana"[:"":True], "invalid slice step: got bool, want int")
# in, not in
assert.true("oo" in "food")
assert.true("ox" not in "food")
assert.true("" in "food")
assert.true("" in "")
assert.fails(lambda: 1 in "", "requires string as left operand")
assert.fails(lambda: "" in 1, "unknown binary op: string in int")
# ==, !=
assert.eq("hello", "he" + "llo")
assert.ne("hello", "Hello")
# hash must follow java.lang.String.hashCode.
wanthash = {
"": 0,
"\0" * 100: 0,
"hello": 99162322,
"world": 113318802,
"Hello, 世界!": 417292677,
}
gothash = {s: hash(s) for s in wanthash}
assert.eq(gothash, wanthash)
# TODO(adonovan): ordered comparisons
# string % tuple formatting
assert.eq("A %d %x Z" % (123, 456), "A 123 1c8 Z")
assert.eq("A %(foo)d %(bar)s Z" % {"foo": 123, "bar": "hi"}, "A 123 hi Z")
assert.eq("%s %r" % ("hi", "hi"), 'hi "hi"') # TODO(adonovan): use ''-quotation
assert.eq("%%d %d" % 1, "%d 1")
assert.fails(lambda: "%d %d" % 1, "not enough arguments for format string")
assert.fails(lambda: "%d %d" % (1, 2, 3), "too many arguments for format string")
assert.fails(lambda: "" % 1, "too many arguments for format string")
# %c
assert.eq("%c" % 65, "A")
assert.eq("%c" % 0x3b1, "α")
assert.eq("%c" % "A", "A")
assert.eq("%c" % "α", "α")
assert.fails(lambda: "%c" % "abc", "requires a single-character string")
assert.fails(lambda: "%c" % "", "requires a single-character string")
assert.fails(lambda: "%c" % 65.0, "requires int or single-character string")
assert.fails(lambda: "%c" % 10000000, "requires a valid Unicode code point")
assert.fails(lambda: "%c" % -1, "requires a valid Unicode code point")
# TODO(adonovan): more tests
# str.format
assert.eq("a{}b".format(123), "a123b")
assert.eq("a{}b{}c{}d{}".format(1, 2, 3, 4), "a1b2c3d4")
assert.eq("a{{b".format(), "a{b")
assert.eq("a}}b".format(), "a}b")
assert.eq("a{{b}}c".format(), "a{b}c")
assert.eq("a{x}b{y}c{}".format(1, x = 2, y = 3), "a2b3c1")
assert.fails(lambda: "a{z}b".format(x = 1), "keyword z not found")
assert.fails(lambda: "{-1}".format(1), "keyword -1 not found")
assert.fails(lambda: "{-0}".format(1), "keyword -0 not found")
assert.fails(lambda: "{+0}".format(1), "keyword \\+0 not found")
assert.fails(lambda: "{+1}".format(1), "keyword \\+1 not found") # starlark-go/issues/114
assert.eq("{0000000000001}".format(0, 1), "1")
assert.eq("{012}".format(*range(100)), "12") # decimal, despite leading zeros
assert.fails(lambda: "{0,1} and {1}".format(1, 2), "keyword 0,1 not found")
assert.fails(lambda: "a{123}b".format(), "tuple index out of range")
assert.fails(lambda: "a{}b{}c".format(1), "tuple index out of range")
assert.eq("a{010}b".format(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10), "a10b") # index is decimal
assert.fails(lambda: "a{}b{1}c".format(1, 2), "cannot switch from automatic field numbering to manual")
assert.eq("a{!s}c".format("b"), "abc")
assert.eq("a{!r}c".format("b"), r'a"b"c')
assert.eq("a{x!r}c".format(x = "b"), r'a"b"c')
assert.fails(lambda: "{x!}".format(x = 1), "unknown conversion")
assert.fails(lambda: "{x!:}".format(x = 1), "unknown conversion")
assert.fails(lambda: "{a.b}".format(1), "syntax x.y is not supported")
assert.fails(lambda: "{a[0]}".format(1), "syntax a\\[i\\] is not supported")
assert.fails(lambda: "{ {} }".format(1), "nested replacement fields not supported")
assert.fails(lambda: "{{}".format(1), "single '}' in format")
assert.fails(lambda: "{}}".format(1), "single '}' in format")
assert.fails(lambda: "}}{".format(1), "unmatched '{' in format")
assert.fails(lambda: "}{{".format(1), "single '}' in format")
# str.split, str.rsplit
assert.eq("a.b.c.d".split("."), ["a", "b", "c", "d"])
assert.eq("a.b.c.d".rsplit("."), ["a", "b", "c", "d"])
assert.eq("a.b.c.d".split(".", -1), ["a", "b", "c", "d"])
assert.eq("a.b.c.d".rsplit(".", -1), ["a", "b", "c", "d"])
assert.eq("a.b.c.d".split(".", 0), ["a.b.c.d"])
assert.eq("a.b.c.d".rsplit(".", 0), ["a.b.c.d"])
assert.eq("a.b.c.d".split(".", 1), ["a", "b.c.d"])
assert.eq("a.b.c.d".rsplit(".", 1), ["a.b.c", "d"])
assert.eq("a.b.c.d".split(".", 2), ["a", "b", "c.d"])
assert.eq("a.b.c.d".rsplit(".", 2), ["a.b", "c", "d"])
assert.eq(" ".split("."), [" "])
assert.eq(" ".rsplit("."), [" "])
# {,r}split on white space:
assert.eq(" a bc\n def \t ghi".split(), ["a", "bc", "def", "ghi"])
assert.eq(" a bc\n def \t ghi".split(None), ["a", "bc", "def", "ghi"])
assert.eq(" a bc\n def \t ghi".split(None, 0), ["a bc\n def \t ghi"])
assert.eq(" a bc\n def \t ghi".rsplit(None, 0), [" a bc\n def \t ghi"])
assert.eq(" a bc\n def \t ghi".split(None, 1), ["a", "bc\n def \t ghi"])
assert.eq(" a bc\n def \t ghi".rsplit(None, 1), [" a bc\n def", "ghi"])
assert.eq(" a bc\n def \t ghi".split(None, 2), ["a", "bc", "def \t ghi"])
assert.eq(" a bc\n def \t ghi".rsplit(None, 2), [" a bc", "def", "ghi"])
assert.eq(" a bc\n def \t ghi".split(None, 3), ["a", "bc", "def", "ghi"])
assert.eq(" a bc\n def \t ghi".rsplit(None, 3), [" a", "bc", "def", "ghi"])
assert.eq(" a bc\n def \t ghi".split(None, 4), ["a", "bc", "def", "ghi"])
assert.eq(" a bc\n def \t ghi".rsplit(None, 4), ["a", "bc", "def", "ghi"])
assert.eq(" a bc\n def \t ghi".rsplit(None, 5), ["a", "bc", "def", "ghi"])
assert.eq(" a bc\n def \t ghi ".split(None, 0), ["a bc\n def \t ghi "])
assert.eq(" a bc\n def \t ghi ".rsplit(None, 0), [" a bc\n def \t ghi"])
assert.eq(" a bc\n def \t ghi ".split(None, 1), ["a", "bc\n def \t ghi "])
assert.eq(" a bc\n def \t ghi ".rsplit(None, 1), [" a bc\n def", "ghi"])
# Observe the algorithmic difference when splitting on spaces versus other delimiters.
assert.eq("--aa--bb--cc--".split("-", 0), ["--aa--bb--cc--"]) # contrast this
assert.eq(" aa bb cc ".split(None, 0), ["aa bb cc "]) # with this
assert.eq("--aa--bb--cc--".rsplit("-", 0), ["--aa--bb--cc--"]) # ditto this
assert.eq(" aa bb cc ".rsplit(None, 0), [" aa bb cc"]) # and this
#
assert.eq("--aa--bb--cc--".split("-", 1), ["", "-aa--bb--cc--"])
assert.eq("--aa--bb--cc--".rsplit("-", 1), ["--aa--bb--cc-", ""])
assert.eq(" aa bb cc ".split(None, 1), ["aa", "bb cc "])
assert.eq(" aa bb cc ".rsplit(None, 1), [" aa bb", "cc"])
#
assert.eq("--aa--bb--cc--".split("-", -1), ["", "", "aa", "", "bb", "", "cc", "", ""])
assert.eq("--aa--bb--cc--".rsplit("-", -1), ["", "", "aa", "", "bb", "", "cc", "", ""])
assert.eq(" aa bb cc ".split(None, -1), ["aa", "bb", "cc"])
assert.eq(" aa bb cc ".rsplit(None, -1), ["aa", "bb", "cc"])
assert.eq(" ".split(None), [])
assert.eq(" ".rsplit(None), [])
assert.eq("localhost:80".rsplit(":", 1)[-1], "80")
# str.splitlines
assert.eq("\nabc\ndef".splitlines(), ["", "abc", "def"])
assert.eq("\nabc\ndef".splitlines(True), ["\n", "abc\n", "def"])
assert.eq("\nabc\ndef\n".splitlines(), ["", "abc", "def"])
assert.eq("\nabc\ndef\n".splitlines(True), ["\n", "abc\n", "def\n"])
assert.eq("".splitlines(), []) #
assert.eq("".splitlines(True), []) #
assert.eq("a".splitlines(), ["a"])
assert.eq("a".splitlines(True), ["a"])
assert.eq("\n".splitlines(), [""])
assert.eq("\n".splitlines(True), ["\n"])
assert.eq("a\n".splitlines(), ["a"])
assert.eq("a\n".splitlines(True), ["a\n"])
assert.eq("a\n\nb".splitlines(), ["a", "", "b"])
assert.eq("a\n\nb".splitlines(True), ["a\n", "\n", "b"])
assert.eq("a\nb\nc".splitlines(), ["a", "b", "c"])
assert.eq("a\nb\nc".splitlines(True), ["a\n", "b\n", "c"])
assert.eq("a\nb\nc\n".splitlines(), ["a", "b", "c"])
assert.eq("a\nb\nc\n".splitlines(True), ["a\n", "b\n", "c\n"])
# str.{,l,r}strip
assert.eq(" \tfoo\n ".strip(), "foo")
assert.eq(" \tfoo\n ".lstrip(), "foo\n ")
assert.eq(" \tfoo\n ".rstrip(), " \tfoo")
assert.eq(" \tfoo\n ".strip(""), "foo")
assert.eq(" \tfoo\n ".lstrip(""), "foo\n ")
assert.eq(" \tfoo\n ".rstrip(""), " \tfoo")
assert.eq("blah.h".strip("b.h"), "la")
assert.eq("blah.h".lstrip("b.h"), "lah.h")
assert.eq("blah.h".rstrip("b.h"), "bla")
# str.count
assert.eq("banana".count("a"), 3)
assert.eq("banana".count("a", 2), 2)
assert.eq("banana".count("a", -4, -2), 1)
assert.eq("banana".count("a", 1, 4), 2)
assert.eq("banana".count("a", 0, -100), 0)
# str.{starts,ends}with
assert.true("foo".endswith("oo"))
assert.true(not "foo".endswith("x"))
assert.true("foo".startswith("fo"))
assert.true(not "foo".startswith("x"))
assert.fails(lambda: "foo".startswith(1), "got int.*want string")
#
assert.true("abc".startswith(("a", "A")))
assert.true("ABC".startswith(("a", "A")))
assert.true(not "ABC".startswith(("b", "B")))
assert.fails(lambda: "123".startswith((1, 2)), "got int, for element 0")
assert.fails(lambda: "123".startswith(["3"]), "got list")
#
assert.true("abc".endswith(("c", "C")))
assert.true("ABC".endswith(("c", "C")))
assert.true(not "ABC".endswith(("b", "B")))
assert.fails(lambda: "123".endswith((1, 2)), "got int, for element 0")
assert.fails(lambda: "123".endswith(["3"]), "got list")
# start/end
assert.true("abc".startswith("bc", 1))
assert.true(not "abc".startswith("b", 999))
assert.true("abc".endswith("ab", None, -1))
assert.true(not "abc".endswith("b", None, -999))
# str.replace
assert.eq("banana".replace("a", "o", 1), "bonana")
assert.eq("banana".replace("a", "o"), "bonono")
# TODO(adonovan): more tests
# str.{,r}find
assert.eq("foofoo".find("oo"), 1)
assert.eq("foofoo".find("ox"), -1)
assert.eq("foofoo".find("oo", 2), 4)
assert.eq("foofoo".rfind("oo"), 4)
assert.eq("foofoo".rfind("ox"), -1)
assert.eq("foofoo".rfind("oo", 1, 4), 1)
assert.eq("foofoo".find(""), 0)
assert.eq("foofoo".rfind(""), 6)
# str.{,r}partition
assert.eq("foo/bar/wiz".partition("/"), ("foo", "/", "bar/wiz"))
assert.eq("foo/bar/wiz".rpartition("/"), ("foo/bar", "/", "wiz"))
assert.eq("foo/bar/wiz".partition("."), ("foo/bar/wiz", "", ""))
assert.eq("foo/bar/wiz".rpartition("."), ("", "", "foo/bar/wiz"))
assert.fails(lambda: "foo/bar/wiz".partition(""), "empty separator")
assert.fails(lambda: "foo/bar/wiz".rpartition(""), "empty separator")
assert.eq("?".join(["foo", "a/b/c.go".rpartition("/")[0]]), "foo?a/b")
# str.is{alpha,...}
def test_predicates():
predicates = ["alnum", "alpha", "digit", "lower", "space", "title", "upper"]
table = {
"Hello, World!": "title",
"hello, world!": "lower",
"base64": "alnum lower",
"HAL-9000": "upper",
"Catch-22": "title",
"": "",
"\n\t\r": "space",
"abc": "alnum alpha lower",
"ABC": "alnum alpha upper",
"123": "alnum digit",
"DŽLJ": "alnum alpha upper",
"DžLj": "alnum alpha",
"Dž Lj": "title",
"džlj": "alnum alpha lower",
}
for str, want in table.items():
got = " ".join([name for name in predicates if getattr(str, "is" + name)()])
if got != want:
assert.fail("%r matched [%s], want [%s]" % (str, got, want))
test_predicates()
# Strings are not iterable.
# ok
assert.eq(len("abc"), 3) # len
assert.true("a" in "abc") # str in str
assert.eq("abc"[1], "b") # indexing
# not ok
def for_string():
for x in "abc":
pass
def args(*args):
return args
assert.fails(lambda: args(*"abc"), "must be iterable, not string") # varargs
assert.fails(lambda: list("abc"), "got string, want iterable") # list(str)
assert.fails(lambda: tuple("abc"), "got string, want iterable") # tuple(str)
assert.fails(lambda: set("abc"), "got string, want iterable") # set(str)
assert.fails(lambda: set() | "abc", "unknown binary op: set | string") # set union
assert.fails(lambda: enumerate("ab"), "got string, want iterable") # enumerate
assert.fails(lambda: sorted("abc"), "got string, want iterable") # sorted
assert.fails(lambda: [].extend("bc"), "got string, want iterable") # list.extend
assert.fails(lambda: ",".join("abc"), "got string, want iterable") # string.join
assert.fails(lambda: dict(["ab"]), "not iterable .*string") # dict
assert.fails(for_string, "string value is not iterable") # for loop
assert.fails(lambda: [x for x in "abc"], "string value is not iterable") # comprehension
assert.fails(lambda: all("abc"), "got string, want iterable") # all
assert.fails(lambda: any("abc"), "got string, want iterable") # any
assert.fails(lambda: reversed("abc"), "got string, want iterable") # reversed
assert.fails(lambda: zip("ab", "cd"), "not iterable: string") # zip
# str.join
assert.eq(",".join([]), "")
assert.eq(",".join(["a"]), "a")
assert.eq(",".join(["a", "b"]), "a,b")
assert.eq(",".join(["a", "b", "c"]), "a,b,c")
assert.eq(",".join(("a", "b", "c")), "a,b,c")
assert.eq("".join(("a", "b", "c")), "abc")
assert.fails(lambda: "".join(None), "got NoneType, want iterable")
assert.fails(lambda: "".join(["one", 2]), "join: in list, want string, got int")
# TODO(adonovan): tests for: {,r}index
# str.capitalize
assert.eq("hElLo, WoRlD!".capitalize(), "Hello, world!")
assert.eq("por qué".capitalize(), "Por qué")
assert.eq("¿Por qué?".capitalize(), "¿por qué?")
# str.lower
assert.eq("hElLo, WoRlD!".lower(), "hello, world!")
assert.eq("por qué".lower(), "por qué")
assert.eq("¿Por qué?".lower(), "¿por qué?")
assert.eq("LJUBOVIĆ".lower(), "ljubović")
assert.true("dženan ljubović".islower())
# str.upper
assert.eq("hElLo, WoRlD!".upper(), "HELLO, WORLD!")
assert.eq("por qué".upper(), "POR QUÉ")
assert.eq("¿Por qué?".upper(), "¿POR QUÉ?")
assert.eq("ljubović".upper(), "LJUBOVIĆ")
assert.true("DŽENAN LJUBOVIĆ".isupper())
# str.title
assert.eq("hElLo, WoRlD!".title(), "Hello, World!")
assert.eq("por qué".title(), "Por Qué")
assert.eq("¿Por qué?".title(), "¿Por Qué?")
assert.eq("ljubović".title(), "Ljubović")
assert.true("Dženan Ljubović".istitle())
assert.true(not "DŽenan LJubović".istitle())
# method spell check
assert.fails(lambda: "".starts_with, "no .starts_with field.*did you mean .startswith")
assert.fails(lambda: "".StartsWith, "no .StartsWith field.*did you mean .startswith")
assert.fails(lambda: "".fin, "no .fin field.*.did you mean .find")
# removesuffix
assert.eq("Apricot".removesuffix("cot"), "Apri")
assert.eq("Apricot".removesuffix("Cot"), "Apricot")
assert.eq("Apricot".removesuffix("t"), "Aprico")
assert.eq("a".removesuffix(""), "a")
assert.eq("".removesuffix(""), "")
assert.eq("".removesuffix("a"), "")
assert.eq("Apricot".removesuffix("co"), "Apricot")
assert.eq("Apricotcot".removesuffix("cot"), "Apricot")
# removeprefix
assert.eq("Apricot".removeprefix("Apr"), "icot")
assert.eq("Apricot".removeprefix("apr"), "Apricot")
assert.eq("Apricot".removeprefix("A"), "pricot")
assert.eq("a".removeprefix(""), "a")
assert.eq("".removeprefix(""), "")
assert.eq("".removeprefix("a"), "")
assert.eq("Apricot".removeprefix("pr"), "Apricot")
assert.eq("AprApricot".removeprefix("Apr"), "Apricot")
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/time.star
Vendored
+165
View File
@@ -0,0 +1,165 @@
# Tests of time module.
load('assert.star', 'assert')
load('time.star', 'time')
assert.true(time.now() > time.parse_time("2021-03-20T00:00:00Z"))
assert.eq(time.parse_time("2020-06-26T17:38:36Z"), time.from_timestamp(1593193116))
assert.eq(time.parse_time("2020-06-26T17:38:36.123456789", format="2006-01-02T15:04:05.999999999"), time.from_timestamp(1593193116, 123456789))
assert.eq(time.parse_time("1970-01-01T00:00:00Z").unix, 0)
assert.eq(time.parse_time("1970-01-01T00:00:00Z").unix_nano, 0)
t = time.parse_time("2000-01-02T03:04:05Z")
assert.eq(t.year, 2000)
assert.eq(t.in_location("US/Eastern"), time.parse_time("2000-01-01T22:04:05-05:00"))
assert.eq(t.in_location("US/Eastern").format("3 04 PM"), "10 04 PM")
assert.eq(t - t, time.parse_duration("0s"))
d1s = time.parse_duration("1s")
assert.eq(d1s - d1s, time.parse_duration("0"))
assert.eq(d1s + d1s, time.parse_duration("2s"))
assert.eq(d1s * 5, time.parse_duration("5s"))
assert.eq(time.parse_duration("0s") + time.parse_duration("3m35s"), time.parse_duration("3m35s"))
d10h = time.parse_duration("10h")
# duration attributes
assert.eq(10.0, d10h.hours)
assert.eq(10*60.0, d10h.minutes)
assert.eq(10*60*60.0, d10h.seconds)
assert.eq(10*60*60*1000, d10h.milliseconds)
assert.eq(10*60*60*1000000, d10h.microseconds)
assert.eq(10*60*60*1000000000, d10h.nanoseconds)
# duration type
assert.eq("time.duration", type(d10h))
# duration str
assert.eq("10h0m0s", str(d10h))
# duration hash
durations = {
d10h: "10h",
d1s: "10s",
}
assert.eq("10h", durations[d10h])
assert.eq("10s", durations[d1s])
# duration == duration
# duration != duration
assert.eq(time.parse_duration("1h"), time.parse_duration("1h"))
assert.ne(time.parse_duration("1h"), time.parse_duration("1m"))
# duration < duration
assert.lt(time.parse_duration("1m"), time.parse_duration("1h"))
assert.true(not time.parse_duration("1h") < time.parse_duration("1h"))
assert.true(not time.parse_duration("1h") < time.parse_duration("1m"))
# duration <= duration
assert.true(time.parse_duration("1m") <= time.parse_duration("1h"))
assert.true(time.parse_duration("1h") <= time.parse_duration("1h"))
assert.true(not time.parse_duration("1h") <= time.parse_duration("1m"))
# duration > duration
assert.true(not time.parse_duration("1m") > time.parse_duration("1h"))
assert.true(not time.parse_duration("1h") > time.parse_duration("1h"))
assert.true(time.parse_duration("1h") > time.parse_duration("1m"))
# duration >= duration
assert.true(not time.parse_duration("1m") >= time.parse_duration("1h"))
assert.true(time.parse_duration("1h") >= time.parse_duration("1h"))
assert.true(time.parse_duration("1h") >= time.parse_duration("1m"))
refTime = time.parse_time("2011-04-22T13:33:48Z")
tenHoursAfterRefTime = time.parse_time("2011-04-22T23:33:48Z")
# duration + duration = duration
assert.eq(d10h + d1s, time.parse_duration("10h01s"))
# duration + time = time
assert.eq(d10h + refTime, tenHoursAfterRefTime)
# duration - duration = duration
assert.eq(d10h - d1s, time.parse_duration("9h59m59s"))
# duration / duration = float
assert.eq(d10h / time.parse_duration("16m"), 37.5)
assert.fails(lambda: d10h / time.parse_duration("0"), "division by zero")
# duration / int = duration
assert.eq(d10h / 20, time.parse_duration("30m"))
assert.fails(lambda: d10h / 0, "division by zero")
# int / duration = error
assert.fails(lambda: 20 / d10h, "unsupported operation")
# duration / float = duration
assert.eq(d10h / 37.5, time.parse_duration("16m"))
assert.fails(lambda: d10h / 0.0, "division by zero")
# duration // duration = int
assert.eq(d10h // time.parse_duration("16m"), 37)
assert.fails(lambda: d10h // time.parse_duration("0"), "division by zero")
# duration * int = duration
assert.eq(d1s * 1000, time.parse_duration("16m40s"))
# int * duration = duration
assert.eq(1000 * d1s, time.parse_duration("16m40s"))
# is_valid_timezone(location)
assert.true(time.is_valid_timezone("UTC"))
assert.true(time.is_valid_timezone("US/Eastern"))
assert.true(not time.is_valid_timezone("UKN"))
# time(year=..., month=..., day=..., hour=..., minute=..., second=..., nanosecond=..., location=...)
assert.fails(lambda: time.time(2009, 6, 12, 12, 6, 10, 99, "US/Eastern"), "unexpected positional argument")
t1 = time.time(year=2009, month=6, day=12, hour=12, minute=6, second=10, nanosecond=99, location="US/Eastern")
assert.eq(t1, time.parse_time("2009-06-12T12:06:10.000000099", format="2006-01-02T15:04:05.999999999", location="US/Eastern"))
assert.eq(time.time(year=2012, month=12, day=31), time.parse_time("2012-12-31T00:00:00Z"))
assert.eq(time.time(year=2009, month=6, day=12, hour=12, minute=6, second=10, nanosecond=99, location="UTC"), time.time(year=2009, month=6, day=12, hour=12, minute=6, second=10, nanosecond=99))
# time attributes
assert.eq(2009, t1.year)
assert.eq(6, t1.month)
assert.eq(12, t1.day)
assert.eq(12, t1.hour)
assert.eq(6, t1.minute)
assert.eq(10, t1.second)
assert.eq(99, t1.nanosecond)
assert.eq(1244822770, t1.unix)
assert.eq(1244822770000000099, t1.unix_nano)
assert.true(not time.parse_time("0001-01-01T00:00:00Z"))
assert.true(time.parse_time("2022-01-01T00:00:00Z"))
# time type
assert.eq("time.time", type(refTime))
# duration str
assert.eq("2011-04-22 13:33:48 +0000 UTC", str(refTime))
# duration hash
times = {
refTime: "refTime",
t1: "t1",
}
assert.eq("refTime", times[refTime])
assert.eq("t1", times[t1])
oneSecondAfterRefTime = time.parse_time("2011-04-22T13:33:49Z")
oneYearAfterRefTime = time.parse_time("2012-04-22T13:33:48Z")
oneYearBeforeRefTime = time.parse_time("2010-04-22T13:33:48Z")
twoYearsBeforeRefTime = time.parse_time("2009-04-22T13:33:48Z")
tenHoursBeforeRefTime = time.parse_time("2011-04-22T03:33:48Z")
# time == time
# time != time
assert.eq(refTime, refTime)
assert.ne(refTime, oneSecondAfterRefTime)
# time < time
assert.lt(oneYearBeforeRefTime, refTime)
assert.true(not oneYearBeforeRefTime < oneYearBeforeRefTime)
assert.true(not oneYearBeforeRefTime < twoYearsBeforeRefTime)
# time <= time
assert.true(oneYearBeforeRefTime <= refTime)
assert.true(oneYearBeforeRefTime <= oneYearBeforeRefTime)
assert.true(not oneYearBeforeRefTime <= twoYearsBeforeRefTime)
# time > time
assert.true(oneYearAfterRefTime > refTime)
assert.true(not refTime > refTime)
assert.true(not oneYearBeforeRefTime > refTime)
# time >= time
assert.true(oneYearAfterRefTime >= refTime)
assert.true(refTime >= refTime)
assert.true(not oneYearBeforeRefTime >= refTime)
# time + duration = time
assert.eq(refTime + d10h, tenHoursAfterRefTime)
# time - duration = time
assert.eq(refTime - d10h, tenHoursBeforeRefTime)
# time - time = duration
assert.eq(refTime - tenHoursBeforeRefTime, d10h)
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/tuple.star
Vendored
+55
View File
@@ -0,0 +1,55 @@
# Tests of Starlark 'tuple'
load("assert.star", "assert")
# literal
assert.eq((), ())
assert.eq((1), 1)
assert.eq((1,), (1,))
assert.ne((1), (1,))
assert.eq((1, 2), (1, 2))
assert.eq((1, 2, 3, 4, 5), (1, 2, 3, 4, 5))
assert.ne((1, 2, 3), (1, 2, 4))
# truth
assert.true((False,))
assert.true((False, False))
assert.true(not ())
# indexing, x[i]
assert.eq(("a", "b")[0], "a")
assert.eq(("a", "b")[1], "b")
# slicing, x[i:j]
assert.eq("abcd"[0:4:1], "abcd")
assert.eq("abcd"[::2], "ac")
assert.eq("abcd"[1::2], "bd")
assert.eq("abcd"[4:0:-1], "dcb")
banana = tuple("banana".elems())
assert.eq(banana[7::-2], tuple("aaa".elems()))
assert.eq(banana[6::-2], tuple("aaa".elems()))
assert.eq(banana[5::-2], tuple("aaa".elems()))
assert.eq(banana[4::-2], tuple("nnb".elems()))
# tuple
assert.eq(tuple(), ())
assert.eq(tuple("abc".elems()), ("a", "b", "c"))
assert.eq(tuple(["a", "b", "c"]), ("a", "b", "c"))
assert.eq(tuple([1]), (1,))
assert.fails(lambda: tuple(1), "got int, want iterable")
# tuple * int, int * tuple
abc = tuple("abc".elems())
assert.eq(abc * 0, ())
assert.eq(abc * -1, ())
assert.eq(abc * 1, abc)
assert.eq(abc * 3, ("a", "b", "c", "a", "b", "c", "a", "b", "c"))
assert.eq(0 * abc, ())
assert.eq(-1 * abc, ())
assert.eq(1 * abc, abc)
assert.eq(3 * abc, ("a", "b", "c", "a", "b", "c", "a", "b", "c"))
assert.fails(lambda: abc * (1000000 * 1000000), "repeat count 1000000000000 too large")
assert.fails(lambda: abc * 1000000 * 1000000, "excessive repeat \\(3000000 \\* 1000000 elements")
# TODO(adonovan): test use of tuple as sequence
# (for loop, comprehension, library functions).
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/testdata/while.star
Vendored
+37
View File
@@ -0,0 +1,37 @@
# Tests of Starlark while statement.
# This is a "chunked" file: each "---" effectively starts a new file.
# option:while
load("assert.star", "assert")
def sum(n):
r = 0
while n > 0:
r += n
n -= 1
return r
def while_break(n):
r = 0
while n > 0:
if n == 5:
break
r += n
n -= 1
return r
def while_continue(n):
r = 0
while n > 0:
if n % 2 == 0:
n -= 1
continue
r += n
n -= 1
return r
assert.eq(sum(5), 5+4+3+2+1)
assert.eq(while_break(10), 40)
assert.eq(while_continue(10), 25)
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/unpack.go
+355
View File
@@ -0,0 +1,355 @@
package starlark
// This file defines the Unpack helper functions used by
// built-in functions to interpret their call arguments.
import (
"fmt"
"log"
"reflect"
"strings"
"go.starlark.net/internal/spell"
)
// An Unpacker defines custom argument unpacking behavior.
// See UnpackArgs.
type Unpacker interface {
Unpack(v Value) error
}
// UnpackArgs unpacks the positional and keyword arguments into the
// supplied parameter variables. pairs is an alternating list of names
// and pointers to variables.
//
// If the variable is a bool, integer, string, *List, *Dict, Callable,
// Iterable, or user-defined implementation of Value,
// UnpackArgs performs the appropriate type check.
// Predeclared Go integer types uses the AsInt check.
//
// If the parameter name ends with "?", it is optional.
//
// If the parameter name ends with "??", it is optional and treats the None value
// as if the argument was absent.
//
// If a parameter is marked optional, then all following parameters are
// implicitly optional where or not they are marked.
//
// If the variable implements Unpacker, its Unpack argument
// is called with the argument value, allowing an application
// to define its own argument validation and conversion.
//
// If the variable implements Value, UnpackArgs may call
// its Type() method while constructing the error message.
//
// Examples:
//
// var (
// a Value
// b = MakeInt(42)
// c Value = starlark.None
// )
//
// // 1. mixed parameters, like def f(a, b=42, c=None).
// err := UnpackArgs("f", args, kwargs, "a", &a, "b?", &b, "c?", &c)
//
// // 2. keyword parameters only, like def f(*, a, b, c=None).
// if len(args) > 0 {
// return fmt.Errorf("f: unexpected positional arguments")
// }
// err := UnpackArgs("f", args, kwargs, "a", &a, "b?", &b, "c?", &c)
//
// // 3. positional parameters only, like def f(a, b=42, c=None, /) in Python 3.8.
// err := UnpackPositionalArgs("f", args, kwargs, 1, &a, &b, &c)
//
// More complex forms such as def f(a, b=42, *args, c, d=123, **kwargs)
// require additional logic, but their need in built-ins is exceedingly rare.
//
// In the examples above, the declaration of b with type Int causes UnpackArgs
// to require that b's argument value, if provided, is also an int.
// To allow arguments of any type, while retaining the default value of 42,
// declare b as a Value:
//
// var b Value = MakeInt(42)
//
// The zero value of a variable of type Value, such as 'a' in the
// examples above, is not a valid Starlark value, so if the parameter is
// optional, the caller must explicitly handle the default case by
// interpreting nil as None or some computed default. The same is true
// for the zero values of variables of type *List, *Dict, Callable, or
// Iterable. For example:
//
// // def myfunc(d=None, e=[], f={})
// var (
// d Value
// e *List
// f *Dict
// )
// err := UnpackArgs("myfunc", args, kwargs, "d?", &d, "e?", &e, "f?", &f)
// if d == nil { d = None; }
// if e == nil { e = new(List); }
// if f == nil { f = new(Dict); }
//
func UnpackArgs(fnname string, args Tuple, kwargs []Tuple, pairs ...interface{}) error {
nparams := len(pairs) / 2
var defined intset
defined.init(nparams)
paramName := func(x interface{}) (name string, skipNone bool) { // (no free variables)
name = x.(string)
if strings.HasSuffix(name, "??") {
name = strings.TrimSuffix(name, "??")
skipNone = true
} else if name[len(name)-1] == '?' {
name = name[:len(name)-1]
}
return name, skipNone
}
// positional arguments
if len(args) > nparams {
return fmt.Errorf("%s: got %d arguments, want at most %d",
fnname, len(args), nparams)
}
for i, arg := range args {
defined.set(i)
name, skipNone := paramName(pairs[2*i])
if skipNone {
if _, isNone := arg.(NoneType); isNone {
continue
}
}
if err := unpackOneArg(arg, pairs[2*i+1]); err != nil {
return fmt.Errorf("%s: for parameter %s: %s", fnname, name, err)
}
}
// keyword arguments
kwloop:
for _, item := range kwargs {
name, arg := item[0].(String), item[1]
for i := 0; i < nparams; i++ {
pName, skipNone := paramName(pairs[2*i])
if pName == string(name) {
// found it
if defined.set(i) {
return fmt.Errorf("%s: got multiple values for keyword argument %s",
fnname, name)
}
if skipNone {
if _, isNone := arg.(NoneType); isNone {
continue kwloop
}
}
ptr := pairs[2*i+1]
if err := unpackOneArg(arg, ptr); err != nil {
return fmt.Errorf("%s: for parameter %s: %s", fnname, name, err)
}
continue kwloop
}
}
err := fmt.Errorf("%s: unexpected keyword argument %s", fnname, name)
names := make([]string, 0, nparams)
for i := 0; i < nparams; i += 2 {
param, _ := paramName(pairs[i])
names = append(names, param)
}
if n := spell.Nearest(string(name), names); n != "" {
err = fmt.Errorf("%s (did you mean %s?)", err.Error(), n)
}
return err
}
// Check that all non-optional parameters are defined.
// (We needn't check the first len(args).)
for i := len(args); i < nparams; i++ {
name := pairs[2*i].(string)
if strings.HasSuffix(name, "?") {
break // optional
}
if !defined.get(i) {
return fmt.Errorf("%s: missing argument for %s", fnname, name)
}
}
return nil
}
// UnpackPositionalArgs unpacks the positional arguments into
// corresponding variables. Each element of vars is a pointer; see
// UnpackArgs for allowed types and conversions.
//
// UnpackPositionalArgs reports an error if the number of arguments is
// less than min or greater than len(vars), if kwargs is nonempty, or if
// any conversion fails.
//
// See UnpackArgs for general comments.
func UnpackPositionalArgs(fnname string, args Tuple, kwargs []Tuple, min int, vars ...interface{}) error {
if len(kwargs) > 0 {
return fmt.Errorf("%s: unexpected keyword arguments", fnname)
}
max := len(vars)
if len(args) < min {
var atleast string
if min < max {
atleast = "at least "
}
return fmt.Errorf("%s: got %d arguments, want %s%d", fnname, len(args), atleast, min)
}
if len(args) > max {
var atmost string
if max > min {
atmost = "at most "
}
return fmt.Errorf("%s: got %d arguments, want %s%d", fnname, len(args), atmost, max)
}
for i, arg := range args {
if err := unpackOneArg(arg, vars[i]); err != nil {
return fmt.Errorf("%s: for parameter %d: %s", fnname, i+1, err)
}
}
return nil
}
func unpackOneArg(v Value, ptr interface{}) error {
// On failure, don't clobber *ptr.
switch ptr := ptr.(type) {
case Unpacker:
return ptr.Unpack(v)
case *Value:
*ptr = v
case *string:
s, ok := AsString(v)
if !ok {
return fmt.Errorf("got %s, want string", v.Type())
}
*ptr = s
case *bool:
b, ok := v.(Bool)
if !ok {
return fmt.Errorf("got %s, want bool", v.Type())
}
*ptr = bool(b)
case *int, *int8, *int16, *int32, *int64,
*uint, *uint8, *uint16, *uint32, *uint64, *uintptr:
return AsInt(v, ptr)
case *float64:
f, ok := v.(Float)
if !ok {
return fmt.Errorf("got %s, want float", v.Type())
}
*ptr = float64(f)
case **List:
list, ok := v.(*List)
if !ok {
return fmt.Errorf("got %s, want list", v.Type())
}
*ptr = list
case **Dict:
dict, ok := v.(*Dict)
if !ok {
return fmt.Errorf("got %s, want dict", v.Type())
}
*ptr = dict
case *Callable:
f, ok := v.(Callable)
if !ok {
return fmt.Errorf("got %s, want callable", v.Type())
}
*ptr = f
case *Iterable:
it, ok := v.(Iterable)
if !ok {
return fmt.Errorf("got %s, want iterable", v.Type())
}
*ptr = it
default:
// v must have type *V, where V is some subtype of starlark.Value.
ptrv := reflect.ValueOf(ptr)
if ptrv.Kind() != reflect.Ptr {
log.Panicf("internal error: not a pointer: %T", ptr)
}
paramVar := ptrv.Elem()
if !reflect.TypeOf(v).AssignableTo(paramVar.Type()) {
// The value is not assignable to the variable.
// Detect a possible bug in the Go program that called Unpack:
// If the variable *ptr is not a subtype of Value,
// no value of v can possibly work.
if !paramVar.Type().AssignableTo(reflect.TypeOf(new(Value)).Elem()) {
log.Panicf("pointer element type does not implement Value: %T", ptr)
}
// Report Starlark dynamic type error.
//
// We prefer the Starlark Value.Type name over
// its Go reflect.Type name, but calling the
// Value.Type method on the variable is not safe
// in general. If the variable is an interface,
// the call will fail. Even if the variable has
// a concrete type, it might not be safe to call
// Type() on a zero instance. Thus we must use
// recover.
// Default to Go reflect.Type name
paramType := paramVar.Type().String()
// Attempt to call Value.Type method.
func() {
defer func() { recover() }()
if typer, _ := paramVar.Interface().(interface{ Type() string }); typer != nil {
paramType = typer.Type()
}
}()
return fmt.Errorf("got %s, want %s", v.Type(), paramType)
}
paramVar.Set(reflect.ValueOf(v))
}
return nil
}
type intset struct {
small uint64 // bitset, used if n < 64
large map[int]bool // set, used if n >= 64
}
func (is *intset) init(n int) {
if n >= 64 {
is.large = make(map[int]bool)
}
}
func (is *intset) set(i int) (prev bool) {
if is.large == nil {
prev = is.small&(1<<uint(i)) != 0
is.small |= 1 << uint(i)
} else {
prev = is.large[i]
is.large[i] = true
}
return
}
func (is *intset) get(i int) bool {
if is.large == nil {
return is.small&(1<<uint(i)) != 0
}
return is.large[i]
}
func (is *intset) len() int {
if is.large == nil {
// Suboptimal, but used only for error reporting.
len := 0
for i := 0; i < 64; i++ {
if is.small&(1<<uint(i)) != 0 {
len++
}
}
return len
}
return len(is.large)
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/value.go
+1613
View File
File diff suppressed because it is too large Load Diff
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlark/value_test.go
+157
View File
@@ -0,0 +1,157 @@
// Copyright 2017 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package starlark_test
// This file defines tests of the Value API.
import (
"fmt"
"testing"
"github.com/google/go-cmp/cmp"
"go.starlark.net/starlark"
)
func TestStringMethod(t *testing.T) {
s := starlark.String("hello")
for i, test := range [][2]string{
// quoted string:
{s.String(), `"hello"`},
{fmt.Sprintf("%s", s), `"hello"`},
{fmt.Sprintf("%+s", s), `"hello"`},
{fmt.Sprintf("%v", s), `"hello"`},
{fmt.Sprintf("%+v", s), `"hello"`},
// unquoted:
{s.GoString(), `hello`},
{fmt.Sprintf("%#v", s), `hello`},
} {
got, want := test[0], test[1]
if got != want {
t.Errorf("#%d: got <<%s>>, want <<%s>>", i, got, want)
}
}
}
func TestListAppend(t *testing.T) {
l := starlark.NewList(nil)
l.Append(starlark.String("hello"))
res, ok := starlark.AsString(l.Index(0))
if !ok {
t.Errorf("failed list.Append() got: %s, want: starlark.String", l.Index(0).Type())
}
if res != "hello" {
t.Errorf("failed list.Append() got: %+v, want: hello", res)
}
}
func TestParamDefault(t *testing.T) {
tests := []struct {
desc string
starFn string
wantDefaults []starlark.Value
}{
{
desc: "function with all required params",
starFn: "all_required",
wantDefaults: []starlark.Value{nil, nil, nil},
},
{
desc: "function with all optional params",
starFn: "all_opt",
wantDefaults: []starlark.Value{
starlark.String("a"),
starlark.None,
starlark.String(""),
},
},
{
desc: "function with required and optional params",
starFn: "mix_required_opt",
wantDefaults: []starlark.Value{
nil,
nil,
starlark.String("c"),
starlark.String("d"),
},
},
{
desc: "function with required, optional, and varargs params",
starFn: "with_varargs",
wantDefaults: []starlark.Value{
nil,
starlark.String("b"),
nil,
},
},
{
desc: "function with required, optional, varargs, and keyword-only params",
starFn: "with_varargs_kwonly",
wantDefaults: []starlark.Value{
nil,
starlark.String("b"),
starlark.String("c"),
nil,
nil,
},
},
{
desc: "function with required, optional, and keyword-only params",
starFn: "with_kwonly",
wantDefaults: []starlark.Value{
nil,
starlark.String("b"),
starlark.String("c"),
nil,
},
},
{
desc: "function with required, optional, and kwargs params",
starFn: "with_kwargs",
wantDefaults: []starlark.Value{
nil,
starlark.String("b"),
starlark.String("c"),
nil,
},
},
{
desc: "function with required, optional, varargs, kw-only, and kwargs params",
starFn: "with_varargs_kwonly_kwargs",
wantDefaults: []starlark.Value{
nil,
starlark.String("b"),
starlark.String("c"),
nil,
starlark.String("e"),
nil,
nil,
},
},
}
for _, tt := range tests {
t.Run(tt.desc, func(t *testing.T) {
thread := &starlark.Thread{}
filename := "testdata/function_param.star"
globals, err := starlark.ExecFile(thread, filename, nil, nil)
if err != nil {
t.Fatalf("ExecFile(%v, %q) failed: %v", thread, filename, err)
}
fn, ok := globals[tt.starFn].(*starlark.Function)
if !ok {
t.Fatalf("value %v is not a Starlark function", globals[tt.starFn])
}
var paramDefaults []starlark.Value
for i := 0; i < fn.NumParams(); i++ {
paramDefaults = append(paramDefaults, fn.ParamDefault(i))
}
if diff := cmp.Diff(tt.wantDefaults, paramDefaults); diff != "" {
t.Errorf("param defaults got diff (-want +got):\n%s", diff)
}
})
}
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlarkjson/json.go
+16
View File
@@ -0,0 +1,16 @@
// Copyright 2020 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package starlarkjson is an alias for go.starlark.net/lib/json to provide
// backwards compatibility
//
// Deprecated: use go.starlark.net/lib/json instead
package starlarkjson // import "go.starlark.net/starlarkjson"
import (
"go.starlark.net/lib/json"
)
// Module is an alias of json.Module for backwards import compatibility
var Module = json.Module
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlarkstruct/module.go
+43
View File
@@ -0,0 +1,43 @@
package starlarkstruct
import (
"fmt"
"go.starlark.net/starlark"
)
// A Module is a named collection of values,
// typically a suite of functions imported by a load statement.
//
// It differs from Struct primarily in that its string representation
// does not enumerate its fields.
type Module struct {
Name string
Members starlark.StringDict
}
var _ starlark.HasAttrs = (*Module)(nil)
func (m *Module) Attr(name string) (starlark.Value, error) { return m.Members[name], nil }
func (m *Module) AttrNames() []string { return m.Members.Keys() }
func (m *Module) Freeze() { m.Members.Freeze() }
func (m *Module) Hash() (uint32, error) { return 0, fmt.Errorf("unhashable: %s", m.Type()) }
func (m *Module) String() string { return fmt.Sprintf("<module %q>", m.Name) }
func (m *Module) Truth() starlark.Bool { return true }
func (m *Module) Type() string { return "module" }
// MakeModule may be used as the implementation of a Starlark built-in
// function, module(name, **kwargs). It returns a new module with the
// specified name and members.
func MakeModule(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var name string
if err := starlark.UnpackPositionalArgs(b.Name(), args, nil, 1, &name); err != nil {
return nil, err
}
members := make(starlark.StringDict, len(kwargs))
for _, kwarg := range kwargs {
k := string(kwarg[0].(starlark.String))
members[k] = kwarg[1]
}
return &Module{name, members}, nil
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlarkstruct/struct.go
+282
View File
@@ -0,0 +1,282 @@
// Copyright 2017 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package starlarkstruct defines the Starlark types 'struct' and
// 'module', both optional language extensions.
//
package starlarkstruct // import "go.starlark.net/starlarkstruct"
// It is tempting to introduce a variant of Struct that is a wrapper
// around a Go struct value, for stronger typing guarantees and more
// efficient and convenient field lookup. However:
// 1) all fields of Starlark structs are optional, so we cannot represent
// them using more specific types such as String, Int, *Depset, and
// *File, as such types give no way to represent missing fields.
// 2) the efficiency gain of direct struct field access is rather
// marginal: finding the index of a field by binary searching on the
// sorted list of field names is quite fast compared to the other
// overheads.
// 3) the gains in compactness and spatial locality are also rather
// marginal: the array behind the []entry slice is (due to field name
// strings) only a factor of 2 larger than the corresponding Go struct
// would be, and, like the Go struct, requires only a single allocation.
import (
"fmt"
"sort"
"strings"
"go.starlark.net/starlark"
"go.starlark.net/syntax"
)
// Make is the implementation of a built-in function that instantiates
// an immutable struct from the specified keyword arguments.
//
// An application can add 'struct' to the Starlark environment like so:
//
// globals := starlark.StringDict{
// "struct": starlark.NewBuiltin("struct", starlarkstruct.Make),
// }
//
func Make(_ *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
if len(args) > 0 {
return nil, fmt.Errorf("struct: unexpected positional arguments")
}
return FromKeywords(Default, kwargs), nil
}
// FromKeywords returns a new struct instance whose fields are specified by the
// key/value pairs in kwargs. (Each kwargs[i][0] must be a starlark.String.)
func FromKeywords(constructor starlark.Value, kwargs []starlark.Tuple) *Struct {
if constructor == nil {
panic("nil constructor")
}
s := &Struct{
constructor: constructor,
entries: make(entries, 0, len(kwargs)),
}
for _, kwarg := range kwargs {
k := string(kwarg[0].(starlark.String))
v := kwarg[1]
s.entries = append(s.entries, entry{k, v})
}
sort.Sort(s.entries)
return s
}
// FromStringDict returns a new struct instance whose elements are those of d.
// The constructor parameter specifies the constructor; use Default for an ordinary struct.
func FromStringDict(constructor starlark.Value, d starlark.StringDict) *Struct {
if constructor == nil {
panic("nil constructor")
}
s := &Struct{
constructor: constructor,
entries: make(entries, 0, len(d)),
}
for k, v := range d {
s.entries = append(s.entries, entry{k, v})
}
sort.Sort(s.entries)
return s
}
// Struct is an immutable Starlark type that maps field names to values.
// It is not iterable and does not support len.
//
// A struct has a constructor, a distinct value that identifies a class
// of structs, and which appears in the struct's string representation.
//
// Operations such as x+y fail if the constructors of the two operands
// are not equal.
//
// The default constructor, Default, is the string "struct", but
// clients may wish to 'brand' structs for their own purposes.
// The constructor value appears in the printed form of the value,
// and is accessible using the Constructor method.
//
// Use Attr to access its fields and AttrNames to enumerate them.
type Struct struct {
constructor starlark.Value
entries entries // sorted by name
}
// Default is the default constructor for structs.
// It is merely the string "struct".
const Default = starlark.String("struct")
type entries []entry
func (a entries) Len() int { return len(a) }
func (a entries) Less(i, j int) bool { return a[i].name < a[j].name }
func (a entries) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
type entry struct {
name string
value starlark.Value
}
var (
_ starlark.HasAttrs = (*Struct)(nil)
_ starlark.HasBinary = (*Struct)(nil)
)
// ToStringDict adds a name/value entry to d for each field of the struct.
func (s *Struct) ToStringDict(d starlark.StringDict) {
for _, e := range s.entries {
d[e.name] = e.value
}
}
func (s *Struct) String() string {
buf := new(strings.Builder)
switch constructor := s.constructor.(type) {
case starlark.String:
// NB: The Java implementation always prints struct
// even for Bazel provider instances.
buf.WriteString(constructor.GoString()) // avoid String()'s quotation
default:
buf.WriteString(s.constructor.String())
}
buf.WriteByte('(')
for i, e := range s.entries {
if i > 0 {
buf.WriteString(", ")
}
buf.WriteString(e.name)
buf.WriteString(" = ")
buf.WriteString(e.value.String())
}
buf.WriteByte(')')
return buf.String()
}
// Constructor returns the constructor used to create this struct.
func (s *Struct) Constructor() starlark.Value { return s.constructor }
func (s *Struct) Type() string { return "struct" }
func (s *Struct) Truth() starlark.Bool { return true } // even when empty
func (s *Struct) Hash() (uint32, error) {
// Same algorithm as Tuple.hash, but with different primes.
var x, m uint32 = 8731, 9839
for _, e := range s.entries {
namehash, _ := starlark.String(e.name).Hash()
x = x ^ 3*namehash
y, err := e.value.Hash()
if err != nil {
return 0, err
}
x = x ^ y*m
m += 7349
}
return x, nil
}
func (s *Struct) Freeze() {
for _, e := range s.entries {
e.value.Freeze()
}
}
func (x *Struct) Binary(op syntax.Token, y starlark.Value, side starlark.Side) (starlark.Value, error) {
if y, ok := y.(*Struct); ok && op == syntax.PLUS {
if side == starlark.Right {
x, y = y, x
}
if eq, err := starlark.Equal(x.constructor, y.constructor); err != nil {
return nil, fmt.Errorf("in %s + %s: error comparing constructors: %v",
x.constructor, y.constructor, err)
} else if !eq {
return nil, fmt.Errorf("cannot add structs of different constructors: %s + %s",
x.constructor, y.constructor)
}
z := make(starlark.StringDict, x.len()+y.len())
for _, e := range x.entries {
z[e.name] = e.value
}
for _, e := range y.entries {
z[e.name] = e.value
}
return FromStringDict(x.constructor, z), nil
}
return nil, nil // unhandled
}
// Attr returns the value of the specified field.
func (s *Struct) Attr(name string) (starlark.Value, error) {
// Binary search the entries.
// This implementation is a specialization of
// sort.Search that avoids dynamic dispatch.
n := len(s.entries)
i, j := 0, n
for i < j {
h := int(uint(i+j) >> 1)
if s.entries[h].name < name {
i = h + 1
} else {
j = h
}
}
if i < n && s.entries[i].name == name {
return s.entries[i].value, nil
}
var ctor string
if s.constructor != Default {
ctor = s.constructor.String() + " "
}
return nil, starlark.NoSuchAttrError(
fmt.Sprintf("%sstruct has no .%s attribute", ctor, name))
}
func (s *Struct) len() int { return len(s.entries) }
// AttrNames returns a new sorted list of the struct fields.
func (s *Struct) AttrNames() []string {
names := make([]string, len(s.entries))
for i, e := range s.entries {
names[i] = e.name
}
return names
}
func (x *Struct) CompareSameType(op syntax.Token, y_ starlark.Value, depth int) (bool, error) {
y := y_.(*Struct)
switch op {
case syntax.EQL:
return structsEqual(x, y, depth)
case syntax.NEQ:
eq, err := structsEqual(x, y, depth)
return !eq, err
default:
return false, fmt.Errorf("%s %s %s not implemented", x.Type(), op, y.Type())
}
}
func structsEqual(x, y *Struct, depth int) (bool, error) {
if x.len() != y.len() {
return false, nil
}
if eq, err := starlark.Equal(x.constructor, y.constructor); err != nil {
return false, fmt.Errorf("error comparing struct constructors %v and %v: %v",
x.constructor, y.constructor, err)
} else if !eq {
return false, nil
}
for i, n := 0, x.len(); i < n; i++ {
if x.entries[i].name != y.entries[i].name {
return false, nil
} else if eq, err := starlark.EqualDepth(x.entries[i].value, y.entries[i].value, depth-1); err != nil {
return false, err
} else if !eq {
return false, nil
}
}
return true, nil
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlarkstruct/struct_test.go
+69
View File
@@ -0,0 +1,69 @@
// Copyright 2018 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package starlarkstruct_test
import (
"fmt"
"path/filepath"
"testing"
"go.starlark.net/starlark"
"go.starlark.net/starlarkstruct"
"go.starlark.net/starlarktest"
)
func Test(t *testing.T) {
testdata := starlarktest.DataFile("starlarkstruct", ".")
thread := &starlark.Thread{Load: load}
starlarktest.SetReporter(thread, t)
filename := filepath.Join(testdata, "testdata/struct.star")
predeclared := starlark.StringDict{
"struct": starlark.NewBuiltin("struct", starlarkstruct.Make),
"gensym": starlark.NewBuiltin("gensym", gensym),
}
if _, err := starlark.ExecFile(thread, filename, nil, predeclared); err != nil {
if err, ok := err.(*starlark.EvalError); ok {
t.Fatal(err.Backtrace())
}
t.Fatal(err)
}
}
// load implements the 'load' operation as used in the evaluator tests.
func load(thread *starlark.Thread, module string) (starlark.StringDict, error) {
if module == "assert.star" {
return starlarktest.LoadAssertModule()
}
return nil, fmt.Errorf("load not implemented")
}
// gensym is a built-in function that generates a unique symbol.
func gensym(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var name string
if err := starlark.UnpackArgs("gensym", args, kwargs, "name", &name); err != nil {
return nil, err
}
return &symbol{name: name}, nil
}
// A symbol is a distinct value that acts as a constructor of "branded"
// struct instances, like a class symbol in Python or a "provider" in Bazel.
type symbol struct{ name string }
var _ starlark.Callable = (*symbol)(nil)
func (sym *symbol) Name() string { return sym.name }
func (sym *symbol) String() string { return sym.name }
func (sym *symbol) Type() string { return "symbol" }
func (sym *symbol) Freeze() {} // immutable
func (sym *symbol) Truth() starlark.Bool { return starlark.True }
func (sym *symbol) Hash() (uint32, error) { return 0, fmt.Errorf("unhashable: %s", sym.Type()) }
func (sym *symbol) CallInternal(thread *starlark.Thread, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
if len(args) > 0 {
return nil, fmt.Errorf("%s: unexpected positional arguments", sym)
}
return starlarkstruct.FromKeywords(sym, kwargs), nil
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlarkstruct/testdata/struct.star
Vendored
+63
View File
@@ -0,0 +1,63 @@
# Tests of Starlark 'struct' extension.
# This is not a standard feature and the Go and Starlark APIs may yet change.
load("assert.star", "assert")
assert.eq(str(struct), "<built-in function struct>")
# struct is a constructor for "unbranded" structs.
s = struct(host = "localhost", port = 80)
assert.eq(s, s)
assert.eq(s, struct(host = "localhost", port = 80))
assert.ne(s, struct(host = "localhost", port = 81))
assert.eq(type(s), "struct")
assert.eq(str(s), 'struct(host = "localhost", port = 80)')
assert.eq(s.host, "localhost")
assert.eq(s.port, 80)
assert.fails(lambda : s.protocol, "struct has no .protocol attribute")
assert.eq(dir(s), ["host", "port"])
# Use gensym to create "branded" struct types.
hostport = gensym(name = "hostport")
assert.eq(type(hostport), "symbol")
assert.eq(str(hostport), "hostport")
# Call the symbol to instantiate a new type.
http = hostport(host = "localhost", port = 80)
assert.eq(type(http), "struct")
assert.eq(str(http), 'hostport(host = "localhost", port = 80)') # includes name of constructor
assert.eq(http, http)
assert.eq(http, hostport(host = "localhost", port = 80))
assert.ne(http, hostport(host = "localhost", port = 443))
assert.eq(http.host, "localhost")
assert.eq(http.port, 80)
assert.fails(lambda : http.protocol, "hostport struct has no .protocol attribute")
person = gensym(name = "person")
bob = person(name = "bob", age = 50)
alice = person(name = "alice", city = "NYC")
assert.ne(http, bob) # different constructor symbols
assert.ne(bob, alice) # different fields
hostport2 = gensym(name = "hostport")
assert.eq(hostport, hostport)
assert.ne(hostport, hostport2) # same name, different symbol
assert.ne(http, hostport2(host = "localhost", port = 80)) # equal fields but different ctor symbols
# dir
assert.eq(dir(alice), ["city", "name"])
assert.eq(dir(bob), ["age", "name"])
assert.eq(dir(http), ["host", "port"])
# hasattr, getattr
assert.true(hasattr(alice, "city"))
assert.eq(hasattr(alice, "ageaa"), False)
assert.eq(getattr(alice, "city"), "NYC")
# +
assert.eq(bob + bob, bob)
assert.eq(bob + alice, person(age = 50, city = "NYC", name = "alice"))
assert.eq(alice + bob, person(age = 50, city = "NYC", name = "bob")) # not commutative! a misfeature
assert.fails(lambda : alice + 1, "struct \\+ int")
assert.eq(http + http, http)
assert.fails(lambda : http + bob, "different constructors: hostport \\+ person")
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlarktest/assert.star
+59
View File
@@ -0,0 +1,59 @@
# Predeclared built-ins for this module:
#
# error(msg): report an error in Go's test framework without halting execution.
# This is distinct from the built-in fail function, which halts execution.
# catch(f): evaluate f() and returns its evaluation error message, if any
# matches(str, pattern): report whether str matches regular expression pattern.
# module(**kwargs): a constructor for a module.
# _freeze(x): freeze the value x and everything reachable from it.
# _floateq(x, y): reports floating point equality (within 1 ULP).
#
# Clients may use these functions to define their own testing abstractions.
_num = ("float", "int")
def _eq(x, y):
if x != y:
if (type(x) == "float" and type(y) in _num or
type(y) == "float" and type(x) in _num):
if not _floateq(float(x), float(y)):
error("floats: %r != %r (delta > 1 ulp)" % (x, y))
else:
error("%r != %r" % (x, y))
def _ne(x, y):
if x == y:
error("%r == %r" % (x, y))
def _true(cond, msg = "assertion failed"):
if not cond:
error(msg)
def _lt(x, y):
if not (x < y):
error("%s is not less than %s" % (x, y))
def _contains(x, y):
if y not in x:
error("%s does not contain %s" % (x, y))
def _fails(f, pattern):
"assert_fails asserts that evaluation of f() fails with the specified error."
msg = catch(f)
if msg == None:
error("evaluation succeeded unexpectedly (want error matching %r)" % pattern)
elif not matches(pattern, msg):
error("regular expression (%s) did not match error (%s)" % (pattern, msg))
freeze = _freeze # an exported global whose value is the built-in freeze function
assert = module(
"assert",
fail = error,
eq = _eq,
ne = _ne,
true = _true,
lt = _lt,
contains = _contains,
fails = _fails,
)
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/starlarktest/starlarktest.go
+175
View File
@@ -0,0 +1,175 @@
// Copyright 2017 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package starlarktest defines utilities for testing Starlark programs.
//
// Clients can call LoadAssertModule to load a module that defines
// several functions useful for testing. See assert.star for its
// definition.
//
// The assert.error function, which reports errors to the current Go
// testing.T, requires that clients call SetReporter(thread, t) before use.
package starlarktest // import "go.starlark.net/starlarktest"
import (
_ "embed"
"fmt"
"math"
"os"
"path/filepath"
"regexp"
"strings"
"sync"
"go.starlark.net/starlark"
"go.starlark.net/starlarkstruct"
)
const localKey = "Reporter"
// A Reporter is a value to which errors may be reported.
// It is satisfied by *testing.T.
type Reporter interface {
Error(args ...interface{})
}
// SetReporter associates an error reporter (such as a testing.T in
// a Go test) with the Starlark thread so that Starlark programs may
// report errors to it.
func SetReporter(thread *starlark.Thread, r Reporter) {
thread.SetLocal(localKey, r)
}
// GetReporter returns the Starlark thread's error reporter.
// It must be preceded by a call to SetReporter.
func GetReporter(thread *starlark.Thread) Reporter {
r, ok := thread.Local(localKey).(Reporter)
if !ok {
panic("internal error: starlarktest.SetReporter was not called")
}
return r
}
var (
once sync.Once
assert starlark.StringDict
//go:embed assert.star
assertFileSrc string
assertErr error
)
// LoadAssertModule loads the assert module.
// It is concurrency-safe and idempotent.
func LoadAssertModule() (starlark.StringDict, error) {
once.Do(func() {
predeclared := starlark.StringDict{
"error": starlark.NewBuiltin("error", error_),
"catch": starlark.NewBuiltin("catch", catch),
"matches": starlark.NewBuiltin("matches", matches),
"module": starlark.NewBuiltin("module", starlarkstruct.MakeModule),
"_freeze": starlark.NewBuiltin("freeze", freeze),
"_floateq": starlark.NewBuiltin("floateq", floateq),
}
thread := new(starlark.Thread)
assert, assertErr = starlark.ExecFile(thread, "assert.star", assertFileSrc, predeclared)
})
return assert, assertErr
}
// catch(f) evaluates f() and returns its evaluation error message
// if it failed or None if it succeeded.
func catch(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var fn starlark.Callable
if err := starlark.UnpackArgs("catch", args, kwargs, "fn", &fn); err != nil {
return nil, err
}
if _, err := starlark.Call(thread, fn, nil, nil); err != nil {
return starlark.String(err.Error()), nil
}
return starlark.None, nil
}
// matches(pattern, str) reports whether string str matches the regular expression pattern.
func matches(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var pattern, str string
if err := starlark.UnpackArgs("matches", args, kwargs, "pattern", &pattern, "str", &str); err != nil {
return nil, err
}
ok, err := regexp.MatchString(pattern, str)
if err != nil {
return nil, fmt.Errorf("matches: %s", err)
}
return starlark.Bool(ok), nil
}
// error(x) reports an error to the Go test framework.
func error_(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
if len(args) != 1 {
return nil, fmt.Errorf("error: got %d arguments, want 1", len(args))
}
buf := new(strings.Builder)
stk := thread.CallStack()
stk.Pop()
fmt.Fprintf(buf, "%sError: ", stk)
if s, ok := starlark.AsString(args[0]); ok {
buf.WriteString(s)
} else {
buf.WriteString(args[0].String())
}
GetReporter(thread).Error(buf.String())
return starlark.None, nil
}
// freeze(x) freezes its operand.
func freeze(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
if len(kwargs) > 0 {
return nil, fmt.Errorf("freeze does not accept keyword arguments")
}
if len(args) != 1 {
return nil, fmt.Errorf("freeze got %d arguments, wants 1", len(args))
}
args[0].Freeze()
return args[0], nil
}
// floateq(x, y) reports whether two floats are within 1 ULP of each other.
func floateq(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var xf, yf starlark.Float
if err := starlark.UnpackPositionalArgs(b.Name(), args, kwargs, 2, &xf, &yf); err != nil {
return nil, err
}
res := false
switch {
case xf == yf:
res = true
case math.IsNaN(float64(xf)):
res = math.IsNaN(float64(yf))
case math.IsNaN(float64(yf)):
// false (non-NaN = Nan)
default:
x := math.Float64bits(float64(xf))
y := math.Float64bits(float64(yf))
res = x == y+1 || y == x+1
}
return starlark.Bool(res), nil
}
// DataFile returns the effective filename of the specified
// test data resource. The function abstracts differences between
// 'go build', under which a test runs in its package directory,
// and Blaze, under which a test runs in the root of the tree.
var DataFile = func(pkgdir, filename string) string {
// Check if we're being run by Bazel and change directories if so.
// TEST_SRCDIR and TEST_WORKSPACE are set by the Bazel test runner, so that makes a decent check
testSrcdir := os.Getenv("TEST_SRCDIR")
testWorkspace := os.Getenv("TEST_WORKSPACE")
if testSrcdir != "" && testWorkspace != "" {
return filepath.Join(testSrcdir, "net_starlark_go", pkgdir, filename)
}
// Under go test, ignore pkgdir, which is the directory of the
// current package relative to the module root.
return filename
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/syntax/grammar.txt
+129
View File
@@ -0,0 +1,129 @@
Grammar of Starlark
==================
File = {Statement | newline} eof .
Statement = DefStmt | IfStmt | ForStmt | WhileStmt | SimpleStmt .
DefStmt = 'def' identifier '(' [Parameters [',']] ')' ':' Suite .
Parameters = Parameter {',' Parameter}.
Parameter = identifier | identifier '=' Test | '*' | '*' identifier | '**' identifier .
IfStmt = 'if' Test ':' Suite {'elif' Test ':' Suite} ['else' ':' Suite] .
ForStmt = 'for' LoopVariables 'in' Expression ':' Suite .
WhileStmt = 'while' Test ':' Suite .
Suite = [newline indent {Statement} outdent] | SimpleStmt .
SimpleStmt = SmallStmt {';' SmallStmt} [';'] '\n' .
# NOTE: '\n' optional at EOF
SmallStmt = ReturnStmt
| BreakStmt | ContinueStmt | PassStmt
| AssignStmt
| ExprStmt
| LoadStmt
.
ReturnStmt = 'return' [Expression] .
BreakStmt = 'break' .
ContinueStmt = 'continue' .
PassStmt = 'pass' .
AssignStmt = Expression ('=' | '+=' | '-=' | '*=' | '/=' | '//=' | '%=' | '&=' | '|=' | '^=' | '<<=' | '>>=') Expression .
ExprStmt = Expression .
LoadStmt = 'load' '(' string {',' [identifier '='] string} [','] ')' .
Test = LambdaExpr
| IfExpr
| PrimaryExpr
| UnaryExpr
| BinaryExpr
.
LambdaExpr = 'lambda' [Parameters] ':' Test .
IfExpr = Test 'if' Test 'else' Test .
PrimaryExpr = Operand
| PrimaryExpr DotSuffix
| PrimaryExpr CallSuffix
| PrimaryExpr SliceSuffix
.
Operand = identifier
| int | float | string
| ListExpr | ListComp
| DictExpr | DictComp
| '(' [Expression [',']] ')'
| ('-' | '+') PrimaryExpr
.
DotSuffix = '.' identifier .
CallSuffix = '(' [Arguments [',']] ')' .
SliceSuffix = '[' [Expression] [':' Test [':' Test]] ']' .
Arguments = Argument {',' Argument} .
Argument = Test | identifier '=' Test | '*' Test | '**' Test .
ListExpr = '[' [Expression [',']] ']' .
ListComp = '[' Test {CompClause} ']'.
DictExpr = '{' [Entries [',']] '}' .
DictComp = '{' Entry {CompClause} '}' .
Entries = Entry {',' Entry} .
Entry = Test ':' Test .
CompClause = 'for' LoopVariables 'in' Test | 'if' Test .
UnaryExpr = 'not' Test .
BinaryExpr = Test {Binop Test} .
Binop = 'or'
| 'and'
| '==' | '!=' | '<' | '>' | '<=' | '>=' | 'in' | 'not' 'in'
| '|'
| '^'
| '&'
| '-' | '+'
| '*' | '%' | '/' | '//'
.
Expression = Test {',' Test} .
# NOTE: trailing comma permitted only when within [...] or (...).
LoopVariables = PrimaryExpr {',' PrimaryExpr} .
# Notation (similar to Go spec):
- lowercase and 'quoted' items are lexical tokens.
- Capitalized names denote grammar productions.
- (...) implies grouping
- x | y means either x or y.
- [x] means x is optional
- {x} means x is repeated zero or more times
- The end of each declaration is marked with a period.
# Tokens
- spaces: newline, eof, indent, outdent.
- identifier.
- literals: string, int, float.
- plus all quoted tokens such as '+=', 'return'.
# Notes:
- Ambiguity is resolved using operator precedence.
- The grammar does not enforce the legal order of params and args,
nor that the first compclause must be a 'for'.
TODO:
- explain how the lexer generates indent, outdent, and newline tokens.
- why is unary NOT separated from unary - and +?
- the grammar is (mostly) in LL(1) style so, for example,
dot expressions are formed suffixes, not complete expressions,
which makes the spec harder to read. Reorganize into non-LL(1) form?
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/syntax/options.go
+63
View File
@@ -0,0 +1,63 @@
// Copyright 2023 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package syntax
import _ "unsafe" // for linkname
// FileOptions specifies various per-file options that affect static
// aspects of an individual file such as parsing, name resolution, and
// code generation. (Options that affect global dynamics are typically
// controlled through [starlark.Thread].)
//
// The zero value of FileOptions is the default behavior.
//
// Many functions in this package come in two versions: the legacy
// standalone function (such as [Parse]) uses [LegacyFileOptions],
// whereas the more recent method (such as [Options.Parse]) honors the
// provided options. The second form is preferred. In other packages,
// the modern version is a standalone function with a leading
// FileOptions parameter and the name suffix "Options", such as
// [starlark.ExecFileOptions].
type FileOptions struct {
// resolver
Set bool // allow references to the 'set' built-in function
While bool // allow 'while' statements
TopLevelControl bool // allow if/for/while statements at top-level
GlobalReassign bool // allow reassignment to top-level names
LoadBindsGlobally bool // load creates global not file-local bindings (deprecated)
// compiler
Recursion bool // disable recursion check for functions in this file
}
// TODO(adonovan): provide a canonical flag parser for FileOptions.
// (And use it in the testdata "options:" strings.)
// LegacyFileOptions returns a new FileOptions containing the current
// values of the resolver package's legacy global variables such as
// [resolve.AllowRecursion], etc.
// These variables may be associated with command-line flags.
func LegacyFileOptions() *FileOptions {
return &FileOptions{
Set: resolverAllowSet,
While: resolverAllowGlobalReassign,
TopLevelControl: resolverAllowGlobalReassign,
GlobalReassign: resolverAllowGlobalReassign,
Recursion: resolverAllowRecursion,
LoadBindsGlobally: resolverLoadBindsGlobally,
}
}
// Access resolver (legacy) flags, if they are linked in; false otherwise.
var (
//go:linkname resolverAllowSet go.starlark.net/resolve.AllowSet
resolverAllowSet bool
//go:linkname resolverAllowGlobalReassign go.starlark.net/resolve.AllowGlobalReassign
resolverAllowGlobalReassign bool
//go:linkname resolverAllowRecursion go.starlark.net/resolve.AllowRecursion
resolverAllowRecursion bool
//go:linkname resolverLoadBindsGlobally go.starlark.net/resolve.LoadBindsGlobally
resolverLoadBindsGlobally bool
)
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/syntax/parse.go
+1057
View File
File diff suppressed because it is too large Load Diff
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/syntax/parse_test.go
+487
View File
@@ -0,0 +1,487 @@
// Copyright 2017 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package syntax_test
import (
"bufio"
"bytes"
"fmt"
"go/build"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"go.starlark.net/internal/chunkedfile"
"go.starlark.net/starlarktest"
"go.starlark.net/syntax"
)
func TestExprParseTrees(t *testing.T) {
for _, test := range []struct {
input, want string
}{
{`print(1)`,
`(CallExpr Fn=print Args=(1))`},
{"print(1)\n",
`(CallExpr Fn=print Args=(1))`},
{`x + 1`,
`(BinaryExpr X=x Op=+ Y=1)`},
{`[x for x in y]`,
`(Comprehension Body=x Clauses=((ForClause Vars=x X=y)))`},
{`[x for x in (a if b else c)]`,
`(Comprehension Body=x Clauses=((ForClause Vars=x X=(ParenExpr X=(CondExpr Cond=b True=a False=c)))))`},
{`x[i].f(42)`,
`(CallExpr Fn=(DotExpr X=(IndexExpr X=x Y=i) Name=f) Args=(42))`},
{`x.f()`,
`(CallExpr Fn=(DotExpr X=x Name=f))`},
{`x+y*z`,
`(BinaryExpr X=x Op=+ Y=(BinaryExpr X=y Op=* Y=z))`},
{`x%y-z`,
`(BinaryExpr X=(BinaryExpr X=x Op=% Y=y) Op=- Y=z)`},
{`a + b not in c`,
`(BinaryExpr X=(BinaryExpr X=a Op=+ Y=b) Op=not in Y=c)`},
{`lambda x, *args, **kwargs: None`,
`(LambdaExpr Params=(x (UnaryExpr Op=* X=args) (UnaryExpr Op=** X=kwargs)) Body=None)`},
{`{"one": 1}`,
`(DictExpr List=((DictEntry Key="one" Value=1)))`},
{`a[i]`,
`(IndexExpr X=a Y=i)`},
{`a[i:]`,
`(SliceExpr X=a Lo=i)`},
{`a[:j]`,
`(SliceExpr X=a Hi=j)`},
{`a[::]`,
`(SliceExpr X=a)`},
{`a[::k]`,
`(SliceExpr X=a Step=k)`},
{`[]`,
`(ListExpr)`},
{`[1]`,
`(ListExpr List=(1))`},
{`[1,]`,
`(ListExpr List=(1))`},
{`[1, 2]`,
`(ListExpr List=(1 2))`},
{`()`,
`(TupleExpr)`},
{`(4,)`,
`(ParenExpr X=(TupleExpr List=(4)))`},
{`(4)`,
`(ParenExpr X=4)`},
{`(4, 5)`,
`(ParenExpr X=(TupleExpr List=(4 5)))`},
{`1, 2, 3`,
`(TupleExpr List=(1 2 3))`},
{`1, 2,`,
`unparenthesized tuple with trailing comma`},
{`{}`,
`(DictExpr)`},
{`{"a": 1}`,
`(DictExpr List=((DictEntry Key="a" Value=1)))`},
{`{"a": 1,}`,
`(DictExpr List=((DictEntry Key="a" Value=1)))`},
{`{"a": 1, "b": 2}`,
`(DictExpr List=((DictEntry Key="a" Value=1) (DictEntry Key="b" Value=2)))`},
{`{x: y for (x, y) in z}`,
`(Comprehension Curly Body=(DictEntry Key=x Value=y) Clauses=((ForClause Vars=(ParenExpr X=(TupleExpr List=(x y))) X=z)))`},
{`{x: y for a in b if c}`,
`(Comprehension Curly Body=(DictEntry Key=x Value=y) Clauses=((ForClause Vars=a X=b) (IfClause Cond=c)))`},
{`-1 + +2`,
`(BinaryExpr X=(UnaryExpr Op=- X=1) Op=+ Y=(UnaryExpr Op=+ X=2))`},
{`"foo" + "bar"`,
`(BinaryExpr X="foo" Op=+ Y="bar")`},
{`-1 * 2`, // prec(unary -) > prec(binary *)
`(BinaryExpr X=(UnaryExpr Op=- X=1) Op=* Y=2)`},
{`-x[i]`, // prec(unary -) < prec(x[i])
`(UnaryExpr Op=- X=(IndexExpr X=x Y=i))`},
{`a | b & c | d`, // prec(|) < prec(&)
`(BinaryExpr X=(BinaryExpr X=a Op=| Y=(BinaryExpr X=b Op=& Y=c)) Op=| Y=d)`},
{`a or b and c or d`,
`(BinaryExpr X=(BinaryExpr X=a Op=or Y=(BinaryExpr X=b Op=and Y=c)) Op=or Y=d)`},
{`a and b or c and d`,
`(BinaryExpr X=(BinaryExpr X=a Op=and Y=b) Op=or Y=(BinaryExpr X=c Op=and Y=d))`},
{`f(1, x=y)`,
`(CallExpr Fn=f Args=(1 (BinaryExpr X=x Op== Y=y)))`},
{`f(*args, **kwargs)`,
`(CallExpr Fn=f Args=((UnaryExpr Op=* X=args) (UnaryExpr Op=** X=kwargs)))`},
{`lambda *args, *, x=1, **kwargs: 0`,
`(LambdaExpr Params=((UnaryExpr Op=* X=args) (UnaryExpr Op=*) (BinaryExpr X=x Op== Y=1) (UnaryExpr Op=** X=kwargs)) Body=0)`},
{`lambda *, a, *b: 0`,
`(LambdaExpr Params=((UnaryExpr Op=*) a (UnaryExpr Op=* X=b)) Body=0)`},
{`a if b else c`,
`(CondExpr Cond=b True=a False=c)`},
{`a and not b`,
`(BinaryExpr X=a Op=and Y=(UnaryExpr Op=not X=b))`},
{`[e for x in y if cond1 if cond2]`,
`(Comprehension Body=e Clauses=((ForClause Vars=x X=y) (IfClause Cond=cond1) (IfClause Cond=cond2)))`}, // github.com/google/skylark/issues/53
} {
e, err := syntax.ParseExpr("foo.star", test.input, 0)
var got string
if err != nil {
got = stripPos(err)
} else {
got = treeString(e)
}
if test.want != got {
t.Errorf("parse `%s` = %s, want %s", test.input, got, test.want)
}
}
}
func TestStmtParseTrees(t *testing.T) {
for _, test := range []struct {
input, want string
}{
{`print(1)`,
`(ExprStmt X=(CallExpr Fn=print Args=(1)))`},
{`return 1, 2`,
`(ReturnStmt Result=(TupleExpr List=(1 2)))`},
{`return`,
`(ReturnStmt)`},
{`for i in "abc": break`,
`(ForStmt Vars=i X="abc" Body=((BranchStmt Token=break)))`},
{`for i in "abc": continue`,
`(ForStmt Vars=i X="abc" Body=((BranchStmt Token=continue)))`},
{`for x, y in z: pass`,
`(ForStmt Vars=(TupleExpr List=(x y)) X=z Body=((BranchStmt Token=pass)))`},
{`if True: pass`,
`(IfStmt Cond=True True=((BranchStmt Token=pass)))`},
{`if True: break`,
`(IfStmt Cond=True True=((BranchStmt Token=break)))`},
{`if True: continue`,
`(IfStmt Cond=True True=((BranchStmt Token=continue)))`},
{`if True: pass
else:
pass`,
`(IfStmt Cond=True True=((BranchStmt Token=pass)) False=((BranchStmt Token=pass)))`},
{"if a: pass\nelif b: pass\nelse: pass",
`(IfStmt Cond=a True=((BranchStmt Token=pass)) False=((IfStmt Cond=b True=((BranchStmt Token=pass)) False=((BranchStmt Token=pass)))))`},
{`x, y = 1, 2`,
`(AssignStmt Op== LHS=(TupleExpr List=(x y)) RHS=(TupleExpr List=(1 2)))`},
{`x[i] = 1`,
`(AssignStmt Op== LHS=(IndexExpr X=x Y=i) RHS=1)`},
{`x.f = 1`,
`(AssignStmt Op== LHS=(DotExpr X=x Name=f) RHS=1)`},
{`(x, y) = 1`,
`(AssignStmt Op== LHS=(ParenExpr X=(TupleExpr List=(x y))) RHS=1)`},
{`load("", "a", b="c")`,
`(LoadStmt Module="" From=(a c) To=(a b))`},
{`if True: load("", "a", b="c")`, // load needn't be at toplevel
`(IfStmt Cond=True True=((LoadStmt Module="" From=(a c) To=(a b))))`},
{`def f(x, *args, **kwargs):
pass`,
`(DefStmt Name=f Params=(x (UnaryExpr Op=* X=args) (UnaryExpr Op=** X=kwargs)) Body=((BranchStmt Token=pass)))`},
{`def f(**kwargs, *args): pass`,
`(DefStmt Name=f Params=((UnaryExpr Op=** X=kwargs) (UnaryExpr Op=* X=args)) Body=((BranchStmt Token=pass)))`},
{`def f(a, b, c=d): pass`,
`(DefStmt Name=f Params=(a b (BinaryExpr X=c Op== Y=d)) Body=((BranchStmt Token=pass)))`},
{`def f(a, b=c, d): pass`,
`(DefStmt Name=f Params=(a (BinaryExpr X=b Op== Y=c) d) Body=((BranchStmt Token=pass)))`}, // TODO(adonovan): fix this
{`def f():
def g():
pass
pass
def h():
pass`,
`(DefStmt Name=f Body=((DefStmt Name=g Body=((BranchStmt Token=pass))) (BranchStmt Token=pass)))`},
{"f();g()",
`(ExprStmt X=(CallExpr Fn=f))`},
{"f();",
`(ExprStmt X=(CallExpr Fn=f))`},
{"f();g()\n",
`(ExprStmt X=(CallExpr Fn=f))`},
{"f();\n",
`(ExprStmt X=(CallExpr Fn=f))`},
} {
f, err := syntax.Parse("foo.star", test.input, 0)
if err != nil {
t.Errorf("parse `%s` failed: %v", test.input, stripPos(err))
continue
}
if got := treeString(f.Stmts[0]); test.want != got {
t.Errorf("parse `%s` = %s, want %s", test.input, got, test.want)
}
}
}
// TestFileParseTrees tests sequences of statements, and particularly
// handling of indentation, newlines, line continuations, and blank lines.
func TestFileParseTrees(t *testing.T) {
for _, test := range []struct {
input, want string
}{
{`x = 1
print(x)`,
`(AssignStmt Op== LHS=x RHS=1)
(ExprStmt X=(CallExpr Fn=print Args=(x)))`},
{"if cond:\n\tpass",
`(IfStmt Cond=cond True=((BranchStmt Token=pass)))`},
{"if cond:\n\tpass\nelse:\n\tpass",
`(IfStmt Cond=cond True=((BranchStmt Token=pass)) False=((BranchStmt Token=pass)))`},
{`def f():
pass
pass
pass`,
`(DefStmt Name=f Body=((BranchStmt Token=pass)))
(BranchStmt Token=pass)
(BranchStmt Token=pass)`},
{`pass; pass`,
`(BranchStmt Token=pass)
(BranchStmt Token=pass)`},
{"pass\npass",
`(BranchStmt Token=pass)
(BranchStmt Token=pass)`},
{"pass\n\npass",
`(BranchStmt Token=pass)
(BranchStmt Token=pass)`},
{`x = (1 +
2)`,
`(AssignStmt Op== LHS=x RHS=(ParenExpr X=(BinaryExpr X=1 Op=+ Y=2)))`},
{`x = 1 \
+ 2`,
`(AssignStmt Op== LHS=x RHS=(BinaryExpr X=1 Op=+ Y=2))`},
} {
f, err := syntax.Parse("foo.star", test.input, 0)
if err != nil {
t.Errorf("parse `%s` failed: %v", test.input, stripPos(err))
continue
}
var buf bytes.Buffer
for i, stmt := range f.Stmts {
if i > 0 {
buf.WriteByte('\n')
}
writeTree(&buf, reflect.ValueOf(stmt))
}
if got := buf.String(); test.want != got {
t.Errorf("parse `%s` = %s, want %s", test.input, got, test.want)
}
}
}
// TestCompoundStmt tests handling of REPL-style compound statements.
func TestCompoundStmt(t *testing.T) {
for _, test := range []struct {
input, want string
}{
// blank lines
{"\n",
``},
{" \n",
``},
{"# comment\n",
``},
// simple statement
{"1\n",
`(ExprStmt X=1)`},
{"print(1)\n",
`(ExprStmt X=(CallExpr Fn=print Args=(1)))`},
{"1;2;3;\n",
`(ExprStmt X=1)(ExprStmt X=2)(ExprStmt X=3)`},
{"f();g()\n",
`(ExprStmt X=(CallExpr Fn=f))(ExprStmt X=(CallExpr Fn=g))`},
{"f();\n",
`(ExprStmt X=(CallExpr Fn=f))`},
{"f(\n\n\n\n\n\n\n)\n",
`(ExprStmt X=(CallExpr Fn=f))`},
// complex statements
{"def f():\n pass\n\n",
`(DefStmt Name=f Body=((BranchStmt Token=pass)))`},
{"if cond:\n pass\n\n",
`(IfStmt Cond=cond True=((BranchStmt Token=pass)))`},
// Even as a 1-liner, the following blank line is required.
{"if cond: pass\n\n",
`(IfStmt Cond=cond True=((BranchStmt Token=pass)))`},
// github.com/google/starlark-go/issues/121
{"a; b; c\n",
`(ExprStmt X=a)(ExprStmt X=b)(ExprStmt X=c)`},
{"a; b c\n",
`invalid syntax`},
} {
// Fake readline input from string.
// The ! suffix, which would cause a parse error,
// tests that the parser doesn't read more than necessary.
sc := bufio.NewScanner(strings.NewReader(test.input + "!"))
readline := func() ([]byte, error) {
if sc.Scan() {
return []byte(sc.Text() + "\n"), nil
}
return nil, sc.Err()
}
var got string
f, err := syntax.ParseCompoundStmt("foo.star", readline)
if err != nil {
got = stripPos(err)
} else {
for _, stmt := range f.Stmts {
got += treeString(stmt)
}
}
if test.want != got {
t.Errorf("parse `%s` = %s, want %s", test.input, got, test.want)
}
}
}
func stripPos(err error) string {
s := err.Error()
if i := strings.Index(s, ": "); i >= 0 {
s = s[i+len(": "):] // strip file:line:col
}
return s
}
// treeString prints a syntax node as a parenthesized tree.
// Idents are printed as foo and Literals as "foo" or 42.
// Structs are printed as (type name=value ...).
// Only non-empty fields are shown.
func treeString(n syntax.Node) string {
var buf bytes.Buffer
writeTree(&buf, reflect.ValueOf(n))
return buf.String()
}
func writeTree(out *bytes.Buffer, x reflect.Value) {
switch x.Kind() {
case reflect.String, reflect.Int, reflect.Bool:
fmt.Fprintf(out, "%v", x.Interface())
case reflect.Ptr, reflect.Interface:
if elem := x.Elem(); elem.Kind() == 0 {
out.WriteString("nil")
} else {
writeTree(out, elem)
}
case reflect.Struct:
switch v := x.Interface().(type) {
case syntax.Literal:
switch v.Token {
case syntax.STRING:
fmt.Fprintf(out, "%q", v.Value)
case syntax.BYTES:
fmt.Fprintf(out, "b%q", v.Value)
case syntax.INT:
fmt.Fprintf(out, "%d", v.Value)
}
return
case syntax.Ident:
out.WriteString(v.Name)
return
}
fmt.Fprintf(out, "(%s", strings.TrimPrefix(x.Type().String(), "syntax."))
for i, n := 0, x.NumField(); i < n; i++ {
f := x.Field(i)
if f.Type() == reflect.TypeOf(syntax.Position{}) {
continue // skip positions
}
name := x.Type().Field(i).Name
if name == "commentsRef" {
continue // skip comments fields
}
if f.Type() == reflect.TypeOf(syntax.Token(0)) {
fmt.Fprintf(out, " %s=%s", name, f.Interface())
continue
}
switch f.Kind() {
case reflect.Slice:
if n := f.Len(); n > 0 {
fmt.Fprintf(out, " %s=(", name)
for i := 0; i < n; i++ {
if i > 0 {
out.WriteByte(' ')
}
writeTree(out, f.Index(i))
}
out.WriteByte(')')
}
continue
case reflect.Ptr, reflect.Interface:
if f.IsNil() {
continue
}
case reflect.Int:
if f.Int() != 0 {
fmt.Fprintf(out, " %s=%d", name, f.Int())
}
continue
case reflect.Bool:
if f.Bool() {
fmt.Fprintf(out, " %s", name)
}
continue
}
fmt.Fprintf(out, " %s=", name)
writeTree(out, f)
}
fmt.Fprintf(out, ")")
default:
fmt.Fprintf(out, "%T", x.Interface())
}
}
func TestParseErrors(t *testing.T) {
filename := starlarktest.DataFile("syntax", "testdata/errors.star")
for _, chunk := range chunkedfile.Read(filename, t) {
_, err := syntax.Parse(filename, chunk.Source, 0)
switch err := err.(type) {
case nil:
// ok
case syntax.Error:
chunk.GotError(int(err.Pos.Line), err.Msg)
default:
t.Error(err)
}
chunk.Done()
}
}
func TestFilePortion(t *testing.T) {
// Imagine that the Starlark file or expression print(x.f) is extracted
// from the middle of a file in some hypothetical template language;
// see https://github.com/google/starlark-go/issues/346. For example:
// --
// {{loop x seq}}
// {{print(x.f)}}
// {{end}}
// --
fp := syntax.FilePortion{Content: []byte("print(x.f)"), FirstLine: 2, FirstCol: 4}
file, err := syntax.Parse("foo.template", fp, 0)
if err != nil {
t.Fatal(err)
}
span := fmt.Sprint(file.Stmts[0].Span())
want := "foo.template:2:4 foo.template:2:14"
if span != want {
t.Errorf("wrong span: got %q, want %q", span, want)
}
}
// dataFile is the same as starlarktest.DataFile.
// We make a copy to avoid a dependency cycle.
var dataFile = func(pkgdir, filename string) string {
return filepath.Join(build.Default.GOPATH, "src/go.starlark.net", pkgdir, filename)
}
func BenchmarkParse(b *testing.B) {
filename := dataFile("syntax", "testdata/scan.star")
b.StopTimer()
data, err := os.ReadFile(filename)
if err != nil {
b.Fatal(err)
}
b.StartTimer()
for i := 0; i < b.N; i++ {
_, err := syntax.Parse(filename, data, 0)
if err != nil {
b.Fatal(err)
}
}
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/syntax/quote.go
+309
View File
@@ -0,0 +1,309 @@
// Copyright 2017 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package syntax
// Starlark quoted string utilities.
import (
"fmt"
"strconv"
"strings"
"unicode"
"unicode/utf8"
)
// unesc maps single-letter chars following \ to their actual values.
var unesc = [256]byte{
'a': '\a',
'b': '\b',
'f': '\f',
'n': '\n',
'r': '\r',
't': '\t',
'v': '\v',
'\\': '\\',
'\'': '\'',
'"': '"',
}
// esc maps escape-worthy bytes to the char that should follow \.
var esc = [256]byte{
'\a': 'a',
'\b': 'b',
'\f': 'f',
'\n': 'n',
'\r': 'r',
'\t': 't',
'\v': 'v',
'\\': '\\',
'\'': '\'',
'"': '"',
}
// unquote unquotes the quoted string, returning the actual
// string value, whether the original was triple-quoted,
// whether it was a byte string, and an error describing invalid input.
func unquote(quoted string) (s string, triple, isByte bool, err error) {
// Check for raw prefix: means don't interpret the inner \.
raw := false
if strings.HasPrefix(quoted, "r") {
raw = true
quoted = quoted[1:]
}
// Check for bytes prefix.
if strings.HasPrefix(quoted, "b") {
isByte = true
quoted = quoted[1:]
}
if len(quoted) < 2 {
err = fmt.Errorf("string literal too short")
return
}
if quoted[0] != '"' && quoted[0] != '\'' || quoted[0] != quoted[len(quoted)-1] {
err = fmt.Errorf("string literal has invalid quotes")
return
}
// Check for triple quoted string.
quote := quoted[0]
if len(quoted) >= 6 && quoted[1] == quote && quoted[2] == quote && quoted[:3] == quoted[len(quoted)-3:] {
triple = true
quoted = quoted[3 : len(quoted)-3]
} else {
quoted = quoted[1 : len(quoted)-1]
}
// Now quoted is the quoted data, but no quotes.
// If we're in raw mode or there are no escapes or
// carriage returns, we're done.
var unquoteChars string
if raw {
unquoteChars = "\r"
} else {
unquoteChars = "\\\r"
}
if !strings.ContainsAny(quoted, unquoteChars) {
s = quoted
return
}
// Otherwise process quoted string.
// Each iteration processes one escape sequence along with the
// plain text leading up to it.
buf := new(strings.Builder)
for {
// Remove prefix before escape sequence.
i := strings.IndexAny(quoted, unquoteChars)
if i < 0 {
i = len(quoted)
}
buf.WriteString(quoted[:i])
quoted = quoted[i:]
if len(quoted) == 0 {
break
}
// Process carriage return.
if quoted[0] == '\r' {
buf.WriteByte('\n')
if len(quoted) > 1 && quoted[1] == '\n' {
quoted = quoted[2:]
} else {
quoted = quoted[1:]
}
continue
}
// Process escape sequence.
if len(quoted) == 1 {
err = fmt.Errorf(`truncated escape sequence \`)
return
}
switch quoted[1] {
default:
// In Starlark, like Go, a backslash must escape something.
// (Python still treats unnecessary backslashes literally,
// but since 3.6 has emitted a deprecation warning.)
err = fmt.Errorf("invalid escape sequence \\%c", quoted[1])
return
case '\n':
// Ignore the escape and the line break.
quoted = quoted[2:]
case 'a', 'b', 'f', 'n', 'r', 't', 'v', '\\', '\'', '"':
// One-char escape.
// Escapes are allowed for both kinds of quotation
// mark, not just the kind in use.
buf.WriteByte(unesc[quoted[1]])
quoted = quoted[2:]
case '0', '1', '2', '3', '4', '5', '6', '7':
// Octal escape, up to 3 digits, \OOO.
n := int(quoted[1] - '0')
quoted = quoted[2:]
for i := 1; i < 3; i++ {
if len(quoted) == 0 || quoted[0] < '0' || '7' < quoted[0] {
break
}
n = n*8 + int(quoted[0]-'0')
quoted = quoted[1:]
}
if !isByte && n > 127 {
err = fmt.Errorf(`non-ASCII octal escape \%o (use \u%04X for the UTF-8 encoding of U+%04X)`, n, n, n)
return
}
if n >= 256 {
// NOTE: Python silently discards the high bit,
// so that '\541' == '\141' == 'a'.
// Let's see if we can avoid doing that in BUILD files.
err = fmt.Errorf(`invalid escape sequence \%03o`, n)
return
}
buf.WriteByte(byte(n))
case 'x':
// Hexadecimal escape, exactly 2 digits, \xXX. [0-127]
if len(quoted) < 4 {
err = fmt.Errorf(`truncated escape sequence %s`, quoted)
return
}
n, err1 := strconv.ParseUint(quoted[2:4], 16, 0)
if err1 != nil {
err = fmt.Errorf(`invalid escape sequence %s`, quoted[:4])
return
}
if !isByte && n > 127 {
err = fmt.Errorf(`non-ASCII hex escape %s (use \u%04X for the UTF-8 encoding of U+%04X)`,
quoted[:4], n, n)
return
}
buf.WriteByte(byte(n))
quoted = quoted[4:]
case 'u', 'U':
// Unicode code point, 4 (\uXXXX) or 8 (\UXXXXXXXX) hex digits.
sz := 6
if quoted[1] == 'U' {
sz = 10
}
if len(quoted) < sz {
err = fmt.Errorf(`truncated escape sequence %s`, quoted)
return
}
n, err1 := strconv.ParseUint(quoted[2:sz], 16, 0)
if err1 != nil {
err = fmt.Errorf(`invalid escape sequence %s`, quoted[:sz])
return
}
if n > unicode.MaxRune {
err = fmt.Errorf(`code point out of range: %s (max \U%08x)`,
quoted[:sz], n)
return
}
// As in Go, surrogates are disallowed.
if 0xD800 <= n && n < 0xE000 {
err = fmt.Errorf(`invalid Unicode code point U+%04X`, n)
return
}
buf.WriteRune(rune(n))
quoted = quoted[sz:]
}
}
s = buf.String()
return
}
// indexByte returns the index of the first instance of b in s, or else -1.
func indexByte(s string, b byte) int {
for i := 0; i < len(s); i++ {
if s[i] == b {
return i
}
}
return -1
}
// Quote returns a Starlark literal that denotes s.
// If b, it returns a bytes literal.
func Quote(s string, b bool) string {
const hex = "0123456789abcdef"
var runeTmp [utf8.UTFMax]byte
buf := make([]byte, 0, 3*len(s)/2)
if b {
buf = append(buf, 'b')
}
buf = append(buf, '"')
for width := 0; len(s) > 0; s = s[width:] {
r := rune(s[0])
width = 1
if r >= utf8.RuneSelf {
r, width = utf8.DecodeRuneInString(s)
}
if width == 1 && r == utf8.RuneError {
// String (!b) literals accept \xXX escapes only for ASCII,
// but we must use them here to represent invalid bytes.
// The result is not a legal literal.
buf = append(buf, `\x`...)
buf = append(buf, hex[s[0]>>4])
buf = append(buf, hex[s[0]&0xF])
continue
}
if r == '"' || r == '\\' { // always backslashed
buf = append(buf, '\\')
buf = append(buf, byte(r))
continue
}
if strconv.IsPrint(r) {
n := utf8.EncodeRune(runeTmp[:], r)
buf = append(buf, runeTmp[:n]...)
continue
}
switch r {
case '\a':
buf = append(buf, `\a`...)
case '\b':
buf = append(buf, `\b`...)
case '\f':
buf = append(buf, `\f`...)
case '\n':
buf = append(buf, `\n`...)
case '\r':
buf = append(buf, `\r`...)
case '\t':
buf = append(buf, `\t`...)
case '\v':
buf = append(buf, `\v`...)
default:
switch {
case r < ' ' || r == 0x7f:
buf = append(buf, `\x`...)
buf = append(buf, hex[byte(r)>>4])
buf = append(buf, hex[byte(r)&0xF])
case r > utf8.MaxRune:
r = 0xFFFD
fallthrough
case r < 0x10000:
buf = append(buf, `\u`...)
for s := 12; s >= 0; s -= 4 {
buf = append(buf, hex[r>>uint(s)&0xF])
}
default:
buf = append(buf, `\U`...)
for s := 28; s >= 0; s -= 4 {
buf = append(buf, hex[r>>uint(s)&0xF])
}
}
}
}
buf = append(buf, '"')
return string(buf)
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/syntax/quote_test.go
+65
View File
@@ -0,0 +1,65 @@
// Copyright 2017 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package syntax
import (
"strings"
"testing"
)
var quoteTests = []struct {
q string // quoted
s string // unquoted (actual string)
std bool // q is standard form for s
}{
{`""`, "", true},
{`''`, "", false},
{`"hello"`, `hello`, true},
{`'hello'`, `hello`, false},
{`"quote\"here"`, `quote"here`, true},
{`'quote"here'`, `quote"here`, false},
{`"quote'here"`, `quote'here`, true},
{`'quote\'here'`, `quote'here`, false},
{`"\a\b\f\n\r\t\v\x00\x7f"`, "\a\b\f\n\r\t\v\000\x7F", true},
{`"\a\b\f\n\r\t\v\x00\x7f"`, "\a\b\f\n\r\t\v\000\x7F", false},
{`"\a\b\f\n\r\t\v\x00\x7f"`, "\a\b\f\n\r\t\v\000\x7F", false},
{`"\a\b\f\n\r\t\v\x00\x7f\"'\\\x03"`, "\a\b\f\n\r\t\v\x00\x7F\"'\\\x03", true},
{`"\a\b\f\n\r\t\v\x00\x7f\"'\\\x03"`, "\a\b\f\n\r\t\v\x00\x7F\"'\\\x03", false},
{`"\a\b\f\n\r\t\v\x00\x7f\"'\\\x03"`, "\a\b\f\n\r\t\v\x00\x7F\"'\\\x03", false},
{`"\a\b\f\n\r\t\v\x00\x7f\"\\\x03"`, "\a\b\f\n\r\t\v\x00\x7F\"\\\x03", false},
{
`"cat $(SRCS) | grep '\\s*ip_block:' | sed -e 's/\\s*ip_block: \"\\([^ ]*\\)\"/ \x27\\1\x27,/g' >> $@; "`,
"cat $(SRCS) | grep '\\s*ip_block:' | sed -e 's/\\s*ip_block: \"\\([^ ]*\\)\"/ '\\1',/g' >> $@; ",
false,
},
{
`"cat $(SRCS) | grep '\\s*ip_block:' | sed -e 's/\\s*ip_block: \"\\([^ ]*\\)\"/ '\\1',/g' >> $@; "`,
"cat $(SRCS) | grep '\\s*ip_block:' | sed -e 's/\\s*ip_block: \"\\([^ ]*\\)\"/ '\\1',/g' >> $@; ",
true,
},
}
func TestQuote(t *testing.T) {
for _, tt := range quoteTests {
if !tt.std {
continue
}
q := Quote(tt.s, false)
if q != tt.q {
t.Errorf("quote(%#q) = %s, want %s", tt.s, q, tt.q)
}
}
}
func TestUnquote(t *testing.T) {
for _, tt := range quoteTests {
s, triple, _, err := unquote(tt.q)
wantTriple := strings.HasPrefix(tt.q, `"""`) || strings.HasPrefix(tt.q, `'''`)
if s != tt.s || triple != wantTriple || err != nil {
t.Errorf("unquote(%s) = %#q, %v, %v want %#q, %v, nil", tt.q, s, triple, err, tt.s, wantTriple)
}
}
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/syntax/scan.go
+1124
View File
File diff suppressed because it is too large Load Diff
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/syntax/scan_test.go
+310
View File
@@ -0,0 +1,310 @@
// Copyright 2017 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package syntax
import (
"bytes"
"fmt"
"go/build"
"os"
"path/filepath"
"strings"
"testing"
)
func scan(src interface{}) (tokens string, err error) {
sc, err := newScanner("foo.star", src, false)
if err != nil {
return "", err
}
defer sc.recover(&err)
var buf bytes.Buffer
var val tokenValue
for {
tok := sc.nextToken(&val)
if buf.Len() > 0 {
buf.WriteByte(' ')
}
switch tok {
case EOF:
buf.WriteString("EOF")
case IDENT:
buf.WriteString(val.raw)
case INT:
if val.bigInt != nil {
fmt.Fprintf(&buf, "%d", val.bigInt)
} else {
fmt.Fprintf(&buf, "%d", val.int)
}
case FLOAT:
fmt.Fprintf(&buf, "%e", val.float)
case STRING, BYTES:
buf.WriteString(Quote(val.string, tok == BYTES))
default:
buf.WriteString(tok.String())
}
if tok == EOF {
break
}
}
return buf.String(), nil
}
func TestScanner(t *testing.T) {
for _, test := range []struct {
input, want string
}{
{``, "EOF"},
{`123`, "123 EOF"},
{`x.y`, "x . y EOF"},
{`chocolate.éclair`, `chocolate . éclair EOF`},
{`123 "foo" hello x.y`, `123 "foo" hello x . y EOF`},
{`print(x)`, "print ( x ) EOF"},
{`print(x); print(y)`, "print ( x ) ; print ( y ) EOF"},
{"\nprint(\n1\n)\n", "print ( 1 ) newline EOF"}, // final \n is at toplevel on non-blank line => token
{`/ // /= //= ///=`, "/ // /= //= // /= EOF"},
{`# hello
print(x)`, "print ( x ) EOF"},
{`# hello
print(1)
cc_binary(name="foo")
def f(x):
return x+1
print(1)
`,
`print ( 1 ) newline ` +
`cc_binary ( name = "foo" ) newline ` +
`def f ( x ) : newline ` +
`indent return x + 1 newline ` +
`outdent print ( 1 ) newline ` +
`EOF`},
// EOF should act line an implicit newline.
{`def f(): pass`,
"def f ( ) : pass EOF"},
{`def f():
pass`,
"def f ( ) : newline indent pass newline outdent EOF"},
{`def f():
pass
# oops`,
"def f ( ) : newline indent pass newline outdent EOF"},
{`def f():
pass \
`,
"def f ( ) : newline indent pass newline outdent EOF"},
{`def f():
pass
`,
"def f ( ) : newline indent pass newline outdent EOF"},
{`pass
pass`, "pass newline pass EOF"}, // consecutive newlines are consolidated
{`def f():
pass
`, "def f ( ) : newline indent pass newline outdent EOF"},
{`def f():
pass
` + "\n", "def f ( ) : newline indent pass newline outdent EOF"},
{"pass", "pass EOF"},
{"pass\n", "pass newline EOF"},
{"pass\n ", "pass newline EOF"},
{"pass\n \n", "pass newline EOF"},
{"if x:\n pass\n ", "if x : newline indent pass newline outdent EOF"},
{`x = 1 + \
2`, `x = 1 + 2 EOF`},
{`x = 'a\nb'`, `x = "a\nb" EOF`},
{`x = r'a\nb'`, `x = "a\\nb" EOF`},
{"x = 'a\\\nb'", `x = "ab" EOF`},
{`x = '\''`, `x = "'" EOF`},
{`x = "\""`, `x = "\"" EOF`},
{`x = r'\''`, `x = "\\'" EOF`},
{`x = '''\''''`, `x = "'" EOF`},
{`x = r'''\''''`, `x = "\\'" EOF`},
{`x = ''''a'b'c'''`, `x = "'a'b'c" EOF`},
{"x = '''a\nb'''", `x = "a\nb" EOF`},
{"x = '''a\rb'''", `x = "a\nb" EOF`},
{"x = '''a\r\nb'''", `x = "a\nb" EOF`},
{"x = '''a\n\rb'''", `x = "a\n\nb" EOF`},
{"x = r'a\\\nb'", `x = "a\\\nb" EOF`},
{"x = r'a\\\rb'", `x = "a\\\nb" EOF`},
{"x = r'a\\\r\nb'", `x = "a\\\nb" EOF`},
{"a\rb", `a newline b EOF`},
{"a\nb", `a newline b EOF`},
{"a\r\nb", `a newline b EOF`},
{"a\n\nb", `a newline b EOF`},
// numbers
{"0", `0 EOF`},
{"00", `0 EOF`},
{"0.", `0.000000e+00 EOF`},
{"0.e1", `0.000000e+00 EOF`},
{".0", `0.000000e+00 EOF`},
{"0.0", `0.000000e+00 EOF`},
{".e1", `. e1 EOF`},
{"1", `1 EOF`},
{"1.", `1.000000e+00 EOF`},
{".1", `1.000000e-01 EOF`},
{".1e1", `1.000000e+00 EOF`},
{".1e+1", `1.000000e+00 EOF`},
{".1e-1", `1.000000e-02 EOF`},
{"1e1", `1.000000e+01 EOF`},
{"1e+1", `1.000000e+01 EOF`},
{"1e-1", `1.000000e-01 EOF`},
{"123", `123 EOF`},
{"123e45", `1.230000e+47 EOF`},
{"999999999999999999999999999999999999999999999999999", `999999999999999999999999999999999999999999999999999 EOF`},
{"12345678901234567890", `12345678901234567890 EOF`},
// hex
{"0xA", `10 EOF`},
{"0xAAG", `170 G EOF`},
{"0xG", `foo.star:1:1: invalid hex literal`},
{"0XA", `10 EOF`},
{"0XG", `foo.star:1:1: invalid hex literal`},
{"0xA.", `10 . EOF`},
{"0xA.e1", `10 . e1 EOF`},
{"0x12345678deadbeef12345678", `5634002672576678570168178296 EOF`},
// binary
{"0b1010", `10 EOF`},
{"0B111101", `61 EOF`},
{"0b3", `foo.star:1:3: invalid binary literal`},
{"0b1010201", `10 201 EOF`},
{"0b1010.01", `10 1.000000e-02 EOF`},
{"0b0000", `0 EOF`},
// octal
{"0o123", `83 EOF`},
{"0o12834", `10 834 EOF`},
{"0o12934", `10 934 EOF`},
{"0o12934.", `10 9.340000e+02 EOF`},
{"0o12934.1", `10 9.341000e+02 EOF`},
{"0o12934e1", `10 9.340000e+03 EOF`},
{"0o123.", `83 . EOF`},
{"0o123.1", `83 1.000000e-01 EOF`},
{"0123", `foo.star:1:5: obsolete form of octal literal; use 0o123`},
{"012834", `foo.star:1:1: invalid int literal`},
{"012934", `foo.star:1:1: invalid int literal`},
{"i = 012934", `foo.star:1:5: invalid int literal`},
// octal escapes in string literals
{`"\037"`, `"\x1f" EOF`},
{`"\377"`, `foo.star:1:1: non-ASCII octal escape \377 (use \u00FF for the UTF-8 encoding of U+00FF)`},
{`"\378"`, `"\x1f8" EOF`}, // = '\37' + '8'
{`"\400"`, `foo.star:1:1: non-ASCII octal escape \400`}, // unlike Python 2 and 3
// hex escapes
{`"\x00\x20\x09\x41\x7e\x7f"`, `"\x00 \tA~\x7f" EOF`}, // DEL is non-printable
{`"\x80"`, `foo.star:1:1: non-ASCII hex escape`},
{`"\xff"`, `foo.star:1:1: non-ASCII hex escape`},
{`"\xFf"`, `foo.star:1:1: non-ASCII hex escape`},
{`"\xF"`, `foo.star:1:1: truncated escape sequence \xF`},
{`"\x"`, `foo.star:1:1: truncated escape sequence \x`},
{`"\xfg"`, `foo.star:1:1: invalid escape sequence \xfg`},
// Unicode escapes
// \uXXXX
{`"\u0400"`, `"Ѐ" EOF`},
{`"\u100"`, `foo.star:1:1: truncated escape sequence \u100`},
{`"\u04000"`, `"Ѐ0" EOF`}, // = U+0400 + '0'
{`"\u100g"`, `foo.star:1:1: invalid escape sequence \u100g`},
{`"\u4E16"`, `"世" EOF`},
{`"\udc00"`, `foo.star:1:1: invalid Unicode code point U+DC00`}, // surrogate
// \UXXXXXXXX
{`"\U00000400"`, `"Ѐ" EOF`},
{`"\U0000400"`, `foo.star:1:1: truncated escape sequence \U0000400`},
{`"\U000004000"`, `"Ѐ0" EOF`}, // = U+0400 + '0'
{`"\U1000000g"`, `foo.star:1:1: invalid escape sequence \U1000000g`},
{`"\U0010FFFF"`, `"\U0010ffff" EOF`},
{`"\U00110000"`, `foo.star:1:1: code point out of range: \U00110000 (max \U00110000)`},
{`"\U0001F63F"`, `"😿" EOF`},
{`"\U0000dc00"`, `foo.star:1:1: invalid Unicode code point U+DC00`}, // surrogate
// backslash escapes
// As in Go, a backslash must escape something.
// (Python started issuing a deprecation warning in 3.6.)
{`"foo\(bar"`, `foo.star:1:1: invalid escape sequence \(`},
{`"\+"`, `foo.star:1:1: invalid escape sequence \+`},
{`"\w"`, `foo.star:1:1: invalid escape sequence \w`},
{`"\""`, `"\"" EOF`},
{`"\'"`, `"'" EOF`},
{`'\w'`, `foo.star:1:1: invalid escape sequence \w`},
{`'\''`, `"'" EOF`},
{`'\"'`, `"\"" EOF`},
{`"""\w"""`, `foo.star:1:1: invalid escape sequence \w`},
{`"""\""""`, `"\"" EOF`},
{`"""\'"""`, `"'" EOF`},
{`'''\w'''`, `foo.star:1:1: invalid escape sequence \w`},
{`'''\''''`, `"'" EOF`},
{`'''\"'''`, `"\"" EOF`},
{`r"\w"`, `"\\w" EOF`},
{`r"\""`, `"\\\"" EOF`},
{`r"\'"`, `"\\'" EOF`},
{`r'\w'`, `"\\w" EOF`},
{`r'\''`, `"\\'" EOF`},
{`r'\"'`, `"\\\"" EOF`},
{`'a\zb'`, `foo.star:1:1: invalid escape sequence \z`},
{`"\o123"`, `foo.star:1:1: invalid escape sequence \o`},
// bytes literals (where they differ from text strings)
{`b"AЀ世😿"`, `b"AЀ世😿`}, // 1-4 byte encodings, literal
{`b"\x41\u0400\u4e16\U0001F63F"`, `b"AЀ世😿"`}, // same, as escapes
{`b"\377\378\x80\xff\xFf"`, `b"\xff\x1f8\x80\xff\xff" EOF`}, // hex/oct escapes allow non-ASCII
{`b"\400"`, `foo.star:1:2: invalid escape sequence \400`},
{`b"\udc00"`, `foo.star:1:2: invalid Unicode code point U+DC00`}, // (same as string)
// floats starting with octal digits
{"012934.", `1.293400e+04 EOF`},
{"012934.1", `1.293410e+04 EOF`},
{"012934e1", `1.293400e+05 EOF`},
{"0123.", `1.230000e+02 EOF`},
{"0123.1", `1.231000e+02 EOF`},
// github.com/google/skylark/issues/16
{"x ! 0", "foo.star:1:3: unexpected input character '!'"},
// github.com/google/starlark-go/issues/80
{"([{<>}])", "( [ { < > } ] ) EOF"},
{"f();", "f ( ) ; EOF"},
// github.com/google/starlark-go/issues/104
{"def f():\n if x:\n pass\n ", `def f ( ) : newline indent if x : newline indent pass newline outdent outdent EOF`},
{`while cond: pass`, "while cond : pass EOF"},
// github.com/google/starlark-go/issues/107
{"~= ~= 5", "~ = ~ = 5 EOF"},
{"0in", "0 in EOF"},
{"0or", "foo.star:1:3: invalid octal literal"},
{"6in", "6 in EOF"},
{"6or", "6 or EOF"},
} {
got, err := scan(test.input)
if err != nil {
got = err.(Error).Error()
}
// Prefix match allows us to truncate errors in expectations.
// Success cases all end in EOF.
if !strings.HasPrefix(got, test.want) {
t.Errorf("scan `%s` = [%s], want [%s]", test.input, got, test.want)
}
}
}
// dataFile is the same as starlarktest.DataFile.
// We make a copy to avoid a dependency cycle.
var dataFile = func(pkgdir, filename string) string {
return filepath.Join(build.Default.GOPATH, "src/go.starlark.net", pkgdir, filename)
}
func BenchmarkScan(b *testing.B) {
filename := dataFile("syntax", "testdata/scan.star")
b.StopTimer()
data, err := os.ReadFile(filename)
if err != nil {
b.Fatal(err)
}
b.StartTimer()
for i := 0; i < b.N; i++ {
sc, err := newScanner(filename, data, false)
if err != nil {
b.Fatal(err)
}
var val tokenValue
for sc.nextToken(&val) != EOF {
}
}
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/syntax/syntax.go
+529
View File
@@ -0,0 +1,529 @@
// Copyright 2017 The Bazel Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package syntax provides a Starlark parser and abstract syntax tree.
package syntax // import "go.starlark.net/syntax"
// A Node is a node in a Starlark syntax tree.
type Node interface {
// Span returns the start and end position of the expression.
Span() (start, end Position)
// Comments returns the comments associated with this node.
// It returns nil if RetainComments was not specified during parsing,
// or if AllocComments was not called.
Comments() *Comments
// AllocComments allocates a new Comments node if there was none.
// This makes possible to add new comments using Comments() method.
AllocComments()
}
// A Comment represents a single # comment.
type Comment struct {
Start Position
Text string // without trailing newline
}
// Comments collects the comments associated with an expression.
type Comments struct {
Before []Comment // whole-line comments before this expression
Suffix []Comment // end-of-line comments after this expression (up to 1)
// For top-level expressions only, After lists whole-line
// comments following the expression.
After []Comment
}
// A commentsRef is a possibly-nil reference to a set of comments.
// A commentsRef is embedded in each type of syntax node,
// and provides its Comments and AllocComments methods.
type commentsRef struct{ ref *Comments }
// Comments returns the comments associated with a syntax node,
// or nil if AllocComments has not yet been called.
func (cr commentsRef) Comments() *Comments { return cr.ref }
// AllocComments enables comments to be associated with a syntax node.
func (cr *commentsRef) AllocComments() {
if cr.ref == nil {
cr.ref = new(Comments)
}
}
// Start returns the start position of the expression.
func Start(n Node) Position {
start, _ := n.Span()
return start
}
// End returns the end position of the expression.
func End(n Node) Position {
_, end := n.Span()
return end
}
// A File represents a Starlark file.
type File struct {
commentsRef
Path string
Stmts []Stmt
Module interface{} // a *resolve.Module, set by resolver
Options *FileOptions
}
func (x *File) Span() (start, end Position) {
if len(x.Stmts) == 0 {
return
}
start, _ = x.Stmts[0].Span()
_, end = x.Stmts[len(x.Stmts)-1].Span()
return start, end
}
// A Stmt is a Starlark statement.
type Stmt interface {
Node
stmt()
}
func (*AssignStmt) stmt() {}
func (*BranchStmt) stmt() {}
func (*DefStmt) stmt() {}
func (*ExprStmt) stmt() {}
func (*ForStmt) stmt() {}
func (*WhileStmt) stmt() {}
func (*IfStmt) stmt() {}
func (*LoadStmt) stmt() {}
func (*ReturnStmt) stmt() {}
// An AssignStmt represents an assignment:
//
// x = 0
// x, y = y, x
// x += 1
type AssignStmt struct {
commentsRef
OpPos Position
Op Token // = EQ | {PLUS,MINUS,STAR,PERCENT}_EQ
LHS Expr
RHS Expr
}
func (x *AssignStmt) Span() (start, end Position) {
start, _ = x.LHS.Span()
_, end = x.RHS.Span()
return
}
// A DefStmt represents a function definition.
type DefStmt struct {
commentsRef
Def Position
Name *Ident
Lparen Position
Params []Expr // param = ident | ident=expr | * | *ident | **ident
Rparen Position
Body []Stmt
Function interface{} // a *resolve.Function, set by resolver
}
func (x *DefStmt) Span() (start, end Position) {
_, end = x.Body[len(x.Body)-1].Span()
return x.Def, end
}
// An ExprStmt is an expression evaluated for side effects.
type ExprStmt struct {
commentsRef
X Expr
}
func (x *ExprStmt) Span() (start, end Position) {
return x.X.Span()
}
// An IfStmt is a conditional: If Cond: True; else: False.
// 'elseif' is desugared into a chain of IfStmts.
type IfStmt struct {
commentsRef
If Position // IF or ELIF
Cond Expr
True []Stmt
ElsePos Position // ELSE or ELIF
False []Stmt // optional
}
func (x *IfStmt) Span() (start, end Position) {
body := x.False
if body == nil {
body = x.True
}
_, end = body[len(body)-1].Span()
return x.If, end
}
// A LoadStmt loads another module and binds names from it:
// load(Module, "x", y="foo").
//
// The AST is slightly unfaithful to the concrete syntax here because
// Starlark's load statement, so that it can be implemented in Python,
// binds some names (like y above) with an identifier and some (like x)
// without. For consistency we create fake identifiers for all the
// strings.
type LoadStmt struct {
commentsRef
Load Position
Module *Literal // a string
From []*Ident // name defined in loading module
To []*Ident // name in loaded module
Rparen Position
}
func (x *LoadStmt) Span() (start, end Position) {
return x.Load, x.Rparen
}
// ModuleName returns the name of the module loaded by this statement.
func (x *LoadStmt) ModuleName() string { return x.Module.Value.(string) }
// A BranchStmt changes the flow of control: break, continue, pass.
type BranchStmt struct {
commentsRef
Token Token // = BREAK | CONTINUE | PASS
TokenPos Position
}
func (x *BranchStmt) Span() (start, end Position) {
return x.TokenPos, x.TokenPos.add(x.Token.String())
}
// A ReturnStmt returns from a function.
type ReturnStmt struct {
commentsRef
Return Position
Result Expr // may be nil
}
func (x *ReturnStmt) Span() (start, end Position) {
if x.Result == nil {
return x.Return, x.Return.add("return")
}
_, end = x.Result.Span()
return x.Return, end
}
// An Expr is a Starlark expression.
type Expr interface {
Node
expr()
}
func (*BinaryExpr) expr() {}
func (*CallExpr) expr() {}
func (*Comprehension) expr() {}
func (*CondExpr) expr() {}
func (*DictEntry) expr() {}
func (*DictExpr) expr() {}
func (*DotExpr) expr() {}
func (*Ident) expr() {}
func (*IndexExpr) expr() {}
func (*LambdaExpr) expr() {}
func (*ListExpr) expr() {}
func (*Literal) expr() {}
func (*ParenExpr) expr() {}
func (*SliceExpr) expr() {}
func (*TupleExpr) expr() {}
func (*UnaryExpr) expr() {}
// An Ident represents an identifier.
type Ident struct {
commentsRef
NamePos Position
Name string
Binding interface{} // a *resolver.Binding, set by resolver
}
func (x *Ident) Span() (start, end Position) {
return x.NamePos, x.NamePos.add(x.Name)
}
// A Literal represents a literal string or number.
type Literal struct {
commentsRef
Token Token // = STRING | BYTES | INT | FLOAT
TokenPos Position
Raw string // uninterpreted text
Value interface{} // = string | int64 | *big.Int | float64
}
func (x *Literal) Span() (start, end Position) {
return x.TokenPos, x.TokenPos.add(x.Raw)
}
// A ParenExpr represents a parenthesized expression: (X).
type ParenExpr struct {
commentsRef
Lparen Position
X Expr
Rparen Position
}
func (x *ParenExpr) Span() (start, end Position) {
return x.Lparen, x.Rparen.add(")")
}
// A CallExpr represents a function call expression: Fn(Args).
type CallExpr struct {
commentsRef
Fn Expr
Lparen Position
Args []Expr // arg = expr | ident=expr | *expr | **expr
Rparen Position
}
func (x *CallExpr) Span() (start, end Position) {
start, _ = x.Fn.Span()
return start, x.Rparen.add(")")
}
// A DotExpr represents a field or method selector: X.Name.
type DotExpr struct {
commentsRef
X Expr
Dot Position
NamePos Position
Name *Ident
}
func (x *DotExpr) Span() (start, end Position) {
start, _ = x.X.Span()
_, end = x.Name.Span()
return
}
// A Comprehension represents a list or dict comprehension:
// [Body for ... if ...] or {Body for ... if ...}
type Comprehension struct {
commentsRef
Curly bool // {x:y for ...} or {x for ...}, not [x for ...]
Lbrack Position
Body Expr
Clauses []Node // = *ForClause | *IfClause
Rbrack Position
}
func (x *Comprehension) Span() (start, end Position) {
return x.Lbrack, x.Rbrack.add("]")
}
// A ForStmt represents a loop: for Vars in X: Body.
type ForStmt struct {
commentsRef
For Position
Vars Expr // name, or tuple of names
X Expr
Body []Stmt
}
func (x *ForStmt) Span() (start, end Position) {
_, end = x.Body[len(x.Body)-1].Span()
return x.For, end
}
// A WhileStmt represents a while loop: while X: Body.
type WhileStmt struct {
commentsRef
While Position
Cond Expr
Body []Stmt
}
func (x *WhileStmt) Span() (start, end Position) {
_, end = x.Body[len(x.Body)-1].Span()
return x.While, end
}
// A ForClause represents a for clause in a list comprehension: for Vars in X.
type ForClause struct {
commentsRef
For Position
Vars Expr // name, or tuple of names
In Position
X Expr
}
func (x *ForClause) Span() (start, end Position) {
_, end = x.X.Span()
return x.For, end
}
// An IfClause represents an if clause in a list comprehension: if Cond.
type IfClause struct {
commentsRef
If Position
Cond Expr
}
func (x *IfClause) Span() (start, end Position) {
_, end = x.Cond.Span()
return x.If, end
}
// A DictExpr represents a dictionary literal: { List }.
type DictExpr struct {
commentsRef
Lbrace Position
List []Expr // all *DictEntrys
Rbrace Position
}
func (x *DictExpr) Span() (start, end Position) {
return x.Lbrace, x.Rbrace.add("}")
}
// A DictEntry represents a dictionary entry: Key: Value.
// Used only within a DictExpr.
type DictEntry struct {
commentsRef
Key Expr
Colon Position
Value Expr
}
func (x *DictEntry) Span() (start, end Position) {
start, _ = x.Key.Span()
_, end = x.Value.Span()
return start, end
}
// A LambdaExpr represents an inline function abstraction.
type LambdaExpr struct {
commentsRef
Lambda Position
Params []Expr // param = ident | ident=expr | * | *ident | **ident
Body Expr
Function interface{} // a *resolve.Function, set by resolver
}
func (x *LambdaExpr) Span() (start, end Position) {
_, end = x.Body.Span()
return x.Lambda, end
}
// A ListExpr represents a list literal: [ List ].
type ListExpr struct {
commentsRef
Lbrack Position
List []Expr
Rbrack Position
}
func (x *ListExpr) Span() (start, end Position) {
return x.Lbrack, x.Rbrack.add("]")
}
// CondExpr represents the conditional: X if COND else ELSE.
type CondExpr struct {
commentsRef
If Position
Cond Expr
True Expr
ElsePos Position
False Expr
}
func (x *CondExpr) Span() (start, end Position) {
start, _ = x.True.Span()
_, end = x.False.Span()
return start, end
}
// A TupleExpr represents a tuple literal: (List).
type TupleExpr struct {
commentsRef
Lparen Position // optional (e.g. in x, y = 0, 1), but required if List is empty
List []Expr
Rparen Position
}
func (x *TupleExpr) Span() (start, end Position) {
if x.Lparen.IsValid() {
return x.Lparen, x.Rparen
} else {
return Start(x.List[0]), End(x.List[len(x.List)-1])
}
}
// A UnaryExpr represents a unary expression: Op X.
//
// As a special case, UnaryOp{Op:Star} may also represent
// the star parameter in def f(*args) or def f(*, x).
type UnaryExpr struct {
commentsRef
OpPos Position
Op Token
X Expr // may be nil if Op==STAR
}
func (x *UnaryExpr) Span() (start, end Position) {
if x.X != nil {
_, end = x.X.Span()
} else {
end = x.OpPos.add("*")
}
return x.OpPos, end
}
// A BinaryExpr represents a binary expression: X Op Y.
//
// As a special case, BinaryExpr{Op:EQ} may also
// represent a named argument in a call f(k=v)
// or a named parameter in a function declaration
// def f(param=default).
type BinaryExpr struct {
commentsRef
X Expr
OpPos Position
Op Token
Y Expr
}
func (x *BinaryExpr) Span() (start, end Position) {
start, _ = x.X.Span()
_, end = x.Y.Span()
return start, end
}
// A SliceExpr represents a slice or substring expression: X[Lo:Hi:Step].
type SliceExpr struct {
commentsRef
X Expr
Lbrack Position
Lo, Hi, Step Expr // all optional
Rbrack Position
}
func (x *SliceExpr) Span() (start, end Position) {
start, _ = x.X.Span()
return start, x.Rbrack
}
// An IndexExpr represents an index expression: X[Y].
type IndexExpr struct {
commentsRef
X Expr
Lbrack Position
Y Expr
Rbrack Position
}
func (x *IndexExpr) Span() (start, end Position) {
start, _ = x.X.Span()
return start, x.Rbrack
}
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/syntax/testdata/errors.star
Vendored
+212
View File
@@ -0,0 +1,212 @@
# Tests of parse errors.
# This is a "chunked" file; each "---" line demarcates a new parser input.
#
# TODO(adonovan): lots more tests.
x = 1 +
2 ### "got newline, want primary expression"
---
_ = *x ### `got '\*', want primary`
---
# trailing comma is ok
def f(a, ): pass
def f(*args, ): pass
def f(**kwargs, ): pass
---
# Parameters are validated later.
def f(**kwargs, *args, *, b=1, a, **kwargs, *args, *, b=1, a):
pass
---
def f(a, *-b, c): # ### `got '-', want ','`
pass
---
def f(**kwargs, *args, b=1, a, **kwargs, *args, b=1, a):
pass
---
def pass(): ### "not an identifier"
pass
---
def f : ### `got ':', want '\('`
---
# trailing comma is ok
f(a, )
f(*args, )
f(**kwargs, )
---
f(a=1, *, b=2) ### `got ',', want primary`
---
_ = {x:y for y in z} # ok
_ = {x for y in z} ### `got for, want ':'`
---
def f():
pass
pass ### `unindent does not match any outer indentation level`
---
def f(): pass
---
# Blank line after pass => outdent.
def f():
pass
---
# No blank line after pass; EOF acts like a newline.
def f():
pass
---
# This is a well known parsing ambiguity in Python.
# Python 2.7 accepts it but Python3 and Starlark reject it.
_ = [x for x in lambda: True, lambda: False if x()] ### "got lambda, want primary"
_ = [x for x in (lambda: True, lambda: False) if x()] # ok in all dialects
---
# Starlark, following Python 3, allows an unparenthesized
# tuple after 'in' only in a for statement but not in a comprehension.
# (Python 2.7 allows both.)
for x in 1, 2, 3:
print(x)
_ = [x for x in 1, 2, 3] ### `got ',', want ']', for, or if`
---
# Unparenthesized tuple is not allowed as operand of 'if' in comprehension.
_ = [a for b in c if 1, 2] ### `got ',', want ']', for, or if`
---
# Lambda is ok though.
_ = [a for b in c if lambda: d] # ok
# But the body of such a lambda may not be a conditional:
_ = [a for b in c if (lambda: d if e else f)] # ok
_ = [a for b in c if lambda: d if e else f] ### "got else, want ']'"
---
# A lambda is not allowed as the operand of a 'for' clause.
_ = [a for b in lambda: c] ### `got lambda, want primary`
---
# Comparison operations are not associative.
_ = (0 == 1) == 2 # ok
_ = 0 == (1 == 2) # ok
_ = 0 == 1 == 2 ### "== does not associate with =="
---
_ = (0 <= i) < n # ok
_ = 0 <= (i < n) # ok
_ = 0 <= i < n ### "<= does not associate with <"
---
_ = (a in b) not in c # ok
_ = a in (b not in c) # ok
_ = a in b not in c ### "in does not associate with not in"
---
# shift/reduce ambiguity is reduced
_ = [x for x in a if b else c] ### `got else, want ']', for, or if`
---
[a for b in c else d] ### `got else, want ']', for, or if`
---
_ = a + b not c ### "got identifier, want in"
---
f(1+2 = 3) ### "keyword argument must have form name=expr"
---
print(1, 2, 3
### `got end of file, want '\)'`
---
_ = a if b ### "conditional expression without else clause"
---
load("") ### "load statement must import at least 1 symbol"
---
load("", 1) ### `load operand must be "name" or localname="name" \(got int literal\)`
---
load("a", "x") # ok
---
load(1, 2) ### "first operand of load statement must be a string literal"
---
load("a", x) ### `load operand must be "x" or x="originalname"`
---
load("a", x2=x) ### `original name of loaded symbol must be quoted: x2="originalname"`
---
# All of these parse.
load("a", "x")
load("a", "x", y2="y")
load("a", x2="x", "y") # => positional-before-named arg check happens later (!)
---
# 'load' is not an identifier
load = 1 ### `got '=', want '\('`
---
# 'load' is not an identifier
f(load()) ### `got load, want primary`
---
# 'load' is not an identifier
def load(): ### `not an identifier`
pass
---
# 'load' is not an identifier
def f(load): ### `not an identifier`
pass
---
# A load statement allows a trailing comma.
load("module", "x",)
---
x = 1 +
2 ### "got newline, want primary expression"
---
def f():
pass
# this used to cause a spurious indentation error
---
print 1 2 ### `got int literal, want newline`
---
# newlines are not allowed in raw string literals
raw = r'a ### `unexpected newline in string`
b'
---
# The parser permits an unparenthesized tuple expression for the first index.
x[1, 2:] # ok
---
# But not if it has a trailing comma.
x[1, 2,:] ### `got ':', want primary`
---
# Trailing tuple commas are permitted only within parens; see b/28867036.
(a, b,) = 1, 2 # ok
c, d = 1, 2 # ok
---
a, b, = 1, 2 ### `unparenthesized tuple with trailing comma`
---
a, b = 1, 2, ### `unparenthesized tuple with trailing comma`
---
# See github.com/google/starlark-go/issues/48
a = max(range(10))) ### `unexpected '\)'`
---
# github.com/google/starlark-go/issues/85
s = "\x-0" ### `invalid escape sequence`
go/pkg/mod/go.starlark.net@v0.0.0-20231101134539-556fd59b42f6/syntax/testdata/scan.star
Vendored
+1329
View File
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More