Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Matrices with varying number of elements are breaking the build output #2045

Closed
dsanader opened this issue May 17, 2017 · 6 comments
Closed

Comments

@dsanader
Copy link

Hello again,

This is a follow up of https://github.com/drone/drone-ui/issues/114.

Matrices with unbalanced number of elements break how the procs are returned in a build :

matrix:
  include:
    - VAR1: a
    - VAR1: a
      VAR2: b

This happens on the latest drone:0.6. All works fine until a variable is added or removed from a set. The database seems to be populated correctly (the pid, ppid tree is OK), but children in the proc struct are not retrieved accordingling. They all end up in the last proc.

@bradrydzewski
Copy link

bradrydzewski commented May 17, 2017

I'm still unable to repeat. Below is the yaml I am testing with:

pipeline:
  foo:
    image: golang:${golang}
    commands:
      - go version


matrix:
  include:
    - golang: 1.7
    - golang: 1.8
      redis: 3.0

The build passes. The api endpoint returns the expected output:

{
  "procs": [
    {
      "id": 515,
      "build_id": 376,
      "pid": 1,
      "ppid": 0,
      "pgid": 1,
      "name": "",
      "state": "success",
      "exit_code": 0,
      "start_time": 1495058442,
      "end_time": 1495058465,
      "environ": {
        "golang": "1.7"
      },
      "children": [
        {
          "id": 516,
          "build_id": 376,
          "pid": 3,
          "ppid": 1,
          "pgid": 3,
          "name": "clone",
          "state": "success",
          "exit_code": 0,
          "start_time": 1495058442,
          "end_time": 1495058444
        },
        {
          "id": 517,
          "build_id": 376,
          "pid": 4,
          "ppid": 1,
          "pgid": 4,
          "name": "foo",
          "state": "success",
          "exit_code": 0,
          "start_time": 1495058444,
          "end_time": 1495058465
        }
      ]
    },
    {
      "id": 518,
      "build_id": 376,
      "pid": 2,
      "ppid": 0,
      "pgid": 2,
      "name": "",
      "state": "success",
      "exit_code": 0,
      "start_time": 1495058465,
      "end_time": 1495058486,
      "environ": {
        "golang": "1.8",
        "redis": "3.0"
      },
      "children": [
        {
          "id": 519,
          "build_id": 376,
          "pid": 5,
          "ppid": 2,
          "pgid": 5,
          "name": "clone",
          "state": "success",
          "exit_code": 0,
          "start_time": 1495058465,
          "end_time": 1495058466
        },
        {
          "id": 520,
          "build_id": 376,
          "pid": 6,
          "ppid": 2,
          "pgid": 6,
          "name": "foo",
          "state": "success",
          "exit_code": 0,
          "start_time": 1495058466,
          "end_time": 1495058486
        }
      ]
    }
  ]
}

Is it possible that the lack of a variable is causing an invalid yaml for one of your axis? And have you updated to the absolute latest version of 0.6 (to ensure you have this fix #2018)

Can you please provide a full yaml that reproduces the problem?

@dsanader
Copy link
Author

dsanader commented May 17, 2017

I've pulled drone/drone:0.6 :

0.6: Pulling from drone/drone
Digest: sha256:ebde06d44255ff7043902dc71572aff8f58425de8409b815adb19a2587f305b5
Status: Image is up to date for drone/drone:0.6

And restarted the server after dropping the database. I've tried both the sqlite and MySQL database backends. Nothing fancy about the installation, it's very much like the documented "docker-compose" setup.

I've reused your yaml example and applied the following sequence to trigger the problem :

 matrix:
   include:
     - golang: 1.7
+      redis: 3.0
     - golang: 1.8
       redis: 3.0
     - golang: 1.7
       redis: 3.0
     - golang: 1.8
-      redis: 3.0

After that none of the builds can be displayed. Only the last element of the proc array gets a children attribute, filled with all the steps of the pipeline (2 x clone & foo).

@bradrydzewski
Copy link

bradrydzewski commented May 17, 2017

I still cannot repeat the error. The changes above work as expected. Can you please provide a full yaml configuration that can be used to repeat the error?

{
  ...
  "procs": [
    {
      "id": 527,
      "build_id": 381,
      "pid": 1,
      "ppid": 0,
      "pgid": 1,
      "name": "",
      "state": "success",
      "exit_code": 0,
      "start_time": 1495065138,
      "end_time": 1495065159,
      "environ": {
        "golang": "1.8",
        "redis": "3.0"
      },
      "children": [
        {
          "id": 528,
          "build_id": 381,
          "pid": 3,
          "ppid": 1,
          "pgid": 3,
          "name": "clone",
          "state": "success",
          "exit_code": 0,
          "start_time": 1495065138,
          "end_time": 1495065139
        },
        {
          "id": 529,
          "build_id": 381,
          "pid": 4,
          "ppid": 1,
          "pgid": 4,
          "name": "foo",
          "state": "success",
          "exit_code": 0,
          "start_time": 1495065139,
          "end_time": 1495065159
        }
      ]
    },
    {
      "id": 530,
      "build_id": 381,
      "pid": 2,
      "ppid": 0,
      "pgid": 2,
      "name": "",
      "state": "success",
      "exit_code": 0,
      "start_time": 1495065159,
      "end_time": 1495065181,
      "environ": {
        "golang": "1.7"
      },
      "children": [
        {
          "id": 531,
          "build_id": 381,
          "pid": 5,
          "ppid": 2,
          "pgid": 5,
          "name": "clone",
          "state": "success",
          "exit_code": 0,
          "start_time": 1495065159,
          "end_time": 1495065160
        },
        {
          "id": 532,
          "build_id": 381,
          "pid": 6,
          "ppid": 2,
          "pgid": 6,
          "name": "foo",
          "state": "success",
          "exit_code": 0,
          "start_time": 1495065160,
          "end_time": 1495065181
        }
      ]
    }
  ]
}

@dsanader
Copy link
Author

I've reused your own example on a test repo...

clone:
  git:
    image: plugins/git
    skip_verify: true

pipeline:
  foo:
    image: golang:${golang}
    commands:
      - go version
      - sleep 20

matrix:
  include:
    - golang: 1.7
      redis: 3.0
    - golang: 1.8

Output :

{  
  "procs": [
    {
      "id": 25,
      "build_id": 5,
      "pid": 1,
      "ppid": 0,
      "pgid": 1,
      "name": "",
      "state": "success",
      "exit_code": 0,
      "start_time": 1495067057,
      "end_time": 1495067082,
      "environ": {
        "golang": "1.7",
        "redis": "3.0"
      }
    },
    {
      "id": 28,
      "build_id": 5,
      "pid": 2,
      "ppid": 0,
      "pgid": 2,
      "name": "",
      "state": "success",
      "exit_code": 0,
      "start_time": 1495067082,
      "end_time": 1495067107,
      "environ": {
        "golang": "1.8"
      },
      "children": [
        {
          "id": 26,
          "build_id": 5,
          "pid": 3,
          "ppid": 1,
          "pgid": 3,
          "name": "git",
          "state": "success",
          "exit_code": 0,
          "start_time": 1495067058,
          "end_time": 1495067060
        },
        {
          "id": 27,
          "build_id": 5,
          "pid": 4,
          "ppid": 1,
          "pgid": 4,
          "name": "foo",
          "state": "success",
          "exit_code": 0,
          "start_time": 1495067060,
          "end_time": 1495067082
        },
        {
          "id": 29,
          "build_id": 5,
          "pid": 5,
          "ppid": 2,
          "pgid": 5,
          "name": "git",
          "state": "success",
          "exit_code": 0,
          "start_time": 1495067082,
          "end_time": 1495067084
        },
        {
          "id": 30,
          "build_id": 5,
          "pid": 6,
          "ppid": 2,
          "pgid": 6,
          "name": "foo",
          "state": "success",
          "exit_code": 0,
          "start_time": 1495067084,
          "end_time": 1495067106
        }
      ]
    }
  ]
}

@dsanader
Copy link
Author

Hello Brad,

I've looked into the Tree func of model/proc.go. I've notice the procs array passed to the function was ordered as displayed above (parent, parent, child, child, child, child), but the Tree function seems to assume that this array is always nicely ordered.

I've tried out the following piece of code to bound child and parent processes relying on their ids.

                nodes  []*Proc
                parent *Proc
        )
+       kids := make([]*Proc, len(procs))
+       copy(kids, procs)
        for _, proc := range procs {
                if proc.PPID == 0 {
                        nodes = append(nodes, proc)
                        parent = proc
-                       continue
-               } else {
-                       parent.Children = append(parent.Children, proc)
+                       for _, child := range kids {
+                               if child.PPID == parent.PID {
+                                       parent.Children = append(parent.Children, child)
+                               }
+                       }
                }
        }
        return nodes

Let me know what you think about this lead.

The way the rows are inserted/updated/returned might me an explanation on why the issue can't be easily reproduced.

@bradrydzewski
Copy link

bradrydzewski commented May 19, 2017

We had this issue in postgres (see #2015) which does not guarantee ordered results. Sqlite and Mysql did not seem to have this issue, but none the less, I've applied the postgres fix to mysql and sqlite with patch 3e45431

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants