https://github.com/JuliaLang/julia
Revision 66a4fa704e652f60dc3ad2506d3bdc5a34deb1c5 authored by Valentin Churavy on 04 April 2024, 15:47:27 UTC, committed by GitHub on 04 April 2024, 15:47:27 UTC
This pull request changes `failure_expanded` from `Dict{String, Any}` to
`Vector{Dict{String, Any}}` to fix a JSON schema issue in the [Bootleg
JSON
writer](https://github.com/JuliaLang/julia/blob/7eb5cb89fb938a1dc67efa3861b25562767a7bbe/test/buildkitetestjson.jl#L13)
🏴‍☠️ which is responsible for producing JSON test results for [Buildkite
Test Analytics](https://buildkite.com/test-analytics).

The [`failure_expanded` attribute of a test
result](https://buildkite.com/docs/test-analytics/importing-json#json-test-results-data-reference-test-result-objects)
is a JSON array of objects, rather than a single object. I believe this
is to support the possibility of multiple failure reasons in a single
test case, although I'm not sure if that's used anywhere.

I believe the associated uploads (batches of up to 5,000 results) are
currently getting a successful HTTP 202 Accepted response, but the
response body will contain an error for each test case that had a
non-array `failure_expanded`, meaning those test cases will be dropped:

```http
HTTP/1.1 202 Accepted
Content-Type: application/json; charset=utf-8
Date: Tue, 12 Mar 2024 13:11:46 GMT
X-Request-Id: a35322f6-9990-4b8e-8895-d62bd6e10935

{
  "id": "55ac3b92-…",
  "run_id": "bfa6de98-…",
  "queued": 4998,
  "skipped": 2,
  "errors": [
    "Validation failed: Failure expanded must be an Array",
    "Validation failed: Failure expanded must be an Array"
  ],
  "run_url": "http://buildkite.com/…"
}
```

Rather than make the Buildkite API more permissive, I figured I'd come
and fix it upstream, and write my first few tiny lines of Julia while
I'm at it 😁

I've verified that the adjusted JSON output it accepted by our ingestion
system.

---

For verification, I added an error to an arbitrarily selected test
(because [workers don't return information about passing/broken tests,
only errors or
failure](https://github.com/JuliaLang/julia/blob/7eb5cb89fb938a1dc67efa3861b25562767a7bbe/test/runtests.jl#L363-L365)):

```patch
diff --git a/test/char.jl b/test/char.jl
index 1d3579013a..582423e8a3 100644
--- a/test/char.jl
+++ b/test/char.jl
@@ -1,6 +1,7 @@
 # This file is a part of Julia. License is MIT: https://julialang.org/license

 @testset "basic properties" begin
+    @test throw(ErrorException("testing Buildkite JSON"))
     @test typemax(Char) == reinterpret(Char, typemax(UInt32))
     @test typemin(Char) == Char(0)
     @test typemax(Char) == reinterpret(Char, 0xffffffff)
```

… and then `CI=true ./julia test/runtests.jl char` which produces
`test/results_1.json`.

Before:

```json
[
  {
    "file_name": "/Users/pda/code/julia/test/char.jl",
    "history": {
      "duration": 2.123565912246704,
      "start_at": 1.710245465232599e9,
      "end_at": 1.710245467356165e9
    },
    "name": "test_error: throw(ErrorException(\"testing Buildkite JSON\"))",
    "location": "/Users/pda/code/julia/test/char.jl:4",
    "failure_reason": "Exception (unexpectedly) thrown during test",
    "scope": "/Overall/char",
    "failure_expanded": {
      "backtrace": [
        " [1] top-level scope",
        "   @ ~/code/julia/test/char.jl:4",
        " [2] macro expansion",
        "   @ ~/code/julia/usr/share/julia/stdlib/v1.12/Test/src/Test.jl:164 [inlined]",
        " [3] macro expansion",
        "   @ ~/code/julia/test/char.jl:4 [inlined]",
        " [4] macro expansion",
        "   @ ~/code/julia/usr/share/julia/stdlib/v1.12/Test/src/Test.jl:164 [inlined]",
        " [5] macro expansion",
        "   @ ~/code/julia/test/char.jl:4 [inlined]"
      ],
      "expanded": [
        "testing Buildkite JSON"
      ]
    },
    "id": "e9272117-d5f5-f542-039b-cfd3d2e8f33a",
    "result": "failed"
  }
]
```

After:

```json
[
  {
    "file_name": "/Users/pda/code/julia/test/char.jl",
    "history": {
      "duration": 2.123565912246704,
      "start_at": 1.710245465232599e9,
      "end_at": 1.710245467356165e9
    },
    "name": "test_error: throw(ErrorException(\"testing Buildkite JSON\"))",
    "location": "/Users/pda/code/julia/test/char.jl:4",
    "failure_reason": "Exception (unexpectedly) thrown during test",
    "scope": "/Overall/char",
    "failure_expanded": [
      {
        "backtrace": [
          " [1] top-level scope",
          "   @ ~/code/julia/test/char.jl:4",
          " [2] macro expansion",
          "   @ ~/code/julia/usr/share/julia/stdlib/v1.12/Test/src/Test.jl:164 [inlined]",
          " [3] macro expansion",
          "   @ ~/code/julia/test/char.jl:4 [inlined]",
          " [4] macro expansion",
          "   @ ~/code/julia/usr/share/julia/stdlib/v1.12/Test/src/Test.jl:164 [inlined]",
          " [5] macro expansion",
          "   @ ~/code/julia/test/char.jl:4 [inlined]"
        ],
        "expanded": [
          "testing Buildkite JSON"
        ]
      }
    ],
    "id": "e9272117-d5f5-f542-039b-cfd3d2e8f33a",
    "result": "failed"
  }
]
```

Diff:

```patch
--- buildkite-before.json	2024-03-12 23:08:26
+++ buildkite-after.json	2024-03-12 23:07:58
@@ -10,23 +10,25 @@
     "location": "/Users/pda/code/julia/test/char.jl:4",
     "failure_reason": "Exception (unexpectedly) thrown during test",
     "scope": "/Overall/char",
-    "failure_expanded": {
-      "backtrace": [
-        " [1] top-level scope",
-        "   @ ~/code/julia/test/char.jl:4",
-        " [2] macro expansion",
-        "   @ ~/code/julia/usr/share/julia/stdlib/v1.12/Test/src/Test.jl:164 [inlined]",
-        " [3] macro expansion",
-        "   @ ~/code/julia/test/char.jl:4 [inlined]",
-        " [4] macro expansion",
-        "   @ ~/code/julia/usr/share/julia/stdlib/v1.12/Test/src/Test.jl:164 [inlined]",
-        " [5] macro expansion",
-        "   @ ~/code/julia/test/char.jl:4 [inlined]"
-      ],
-      "expanded": [
-        "testing Buildkite JSON"
-      ]
-    },
+    "failure_expanded": [
+      {
+        "backtrace": [
+          " [1] top-level scope",
+          "   @ ~/code/julia/test/char.jl:4",
+          " [2] macro expansion",
+          "   @ ~/code/julia/usr/share/julia/stdlib/v1.12/Test/src/Test.jl:164 [inlined]",
+          " [3] macro expansion",
+          "   @ ~/code/julia/test/char.jl:4 [inlined]",
+          " [4] macro expansion",
+          "   @ ~/code/julia/usr/share/julia/stdlib/v1.12/Test/src/Test.jl:164 [inlined]",
+          " [5] macro expansion",
+          "   @ ~/code/julia/test/char.jl:4 [inlined]"
+        ],
+        "expanded": [
+          "testing Buildkite JSON"
+        ]
+      }
+    ],
     "id": "e9272117-d5f5-f542-039b-cfd3d2e8f33a",
     "result": "failed"
   }
```
2 parent s c371e4c + 1256e3e
History
Tip revision: 66a4fa704e652f60dc3ad2506d3bdc5a34deb1c5 authored by Valentin Churavy on 04 April 2024, 15:47:27 UTC
Buildkite Test Analytics: fix `failure_expanded` (#53706)
Tip revision: 66a4fa7
File Mode Size
.devcontainer
.github
base
cli
contrib
deps
doc
etc
src
stdlib
test
.buildkite-external-version -rw-r--r-- 5 bytes
.clang-format -rw-r--r-- 3.3 KB
.clangd -rw-r--r-- 114 bytes
.codecov.yml -rw-r--r-- 52 bytes
.git-blame-ignore-revs -rw-r--r-- 371 bytes
.gitattributes -rw-r--r-- 65 bytes
.gitignore -rw-r--r-- 571 bytes
.mailmap -rw-r--r-- 12.7 KB
CITATION.bib -rw-r--r-- 513 bytes
CITATION.cff -rw-r--r-- 1012 bytes
CONTRIBUTING.md -rw-r--r-- 23.4 KB
HISTORY.md -rw-r--r-- 388.1 KB
LICENSE.md -rw-r--r-- 1.3 KB
Make.inc -rw-r--r-- 56.4 KB
Makefile -rw-r--r-- 30.4 KB
NEWS.md -rw-r--r-- 4.4 KB
README.md -rw-r--r-- 7.4 KB
THIRDPARTY.md -rw-r--r-- 3.9 KB
VERSION -rw-r--r-- 11 bytes
julia.spdx.json -rw-r--r-- 37.8 KB
pkgimage.mk -rw-r--r-- 1.4 KB
sysimage.mk -rw-r--r-- 4.2 KB
typos.toml -rw-r--r-- 78 bytes

README.md

back to top