Packages update traverse algorithm optimization

[jorgetutor]

The current approach of updating packages is very useful (from direct packages) as we get commits associated to the root package that allows us to update everything in a predictable timeframe.

The cons of it is that it iterates over several components that does not have an updated (it or its dependencies), if the projects are huge this time is high. I would like to suggest a different approach keeping the same principle. Steps:

1. outdated: Get the full list of outdated packages like composer outdated --lock
2. outdated (filtered): (If possible, not mandatory at this stage) Filter the packages that are not possible to update because the composer.json constraints e.g. 2.0.0 is available but the package constraint is 1.0.x
3. outdatedDirect: Find the "root repository" of each package like composer why done multiple times until finding the "origin"
4. Now we have the list of direct packages start the loop and process like always

I think this could be feasible @omarlopesino but you know the composer API better than I do.
I think this could solve other open issues like #24 #5

Note for the --security flag, we will use composer audit instead as the first step

In case a package is required by several ones we can use the first one found, this can be optimized later

What do you think?

[omarlopesino]

I would like to explore option 3 because I think composer depends/why can show us the direct package of an outdated dependency through the --recursive parameter.

Example:

If we get the first columns of the output we can filter only one of the direct dependencies and add them to the list of modules to update. Then we could only update direct modules having outdated dependencies.

The 'composer depends' command is pretty quick, so I think it would be optimal.

[omarlopesino]

I've worked on this and I found one problem: the recursive parameter sometimes takes too long to finish. I think we can solve this by not using the recursive parameter, use only composer why. Then, if no direct package is found, add the not direct package to the list of packages to updated. This would be optimal and would reduce the list of packages to update.

[omarlopesino]

Here is a proof of concept of the code that would detect direct packages depending on packages needing update:

<?php

$direct_packages = array_filter(explode("\n", shell_exec('composer show --locked --direct --name-only 2>/dev/null | grep -v abandoned')));
$outdated = array_filter(explode("\n", shell_exec('composer show --locked --outdated --name-only 2>/dev/null | grep -v abandoned | grep -v symfony/deprecation-contracts')));

$direct_outdated = [];

$packages_to_update = array_intersect($outdated, $direct_packages);

$outdated_not_direct = array_diff($outdated, $direct_packages);

foreach ($outdated_not_direct as $package) {
  $why = array_filter(explode("\n", shell_exec(sprintf("composer why %s --locked | awk '{print $1}'", $package))));
  foreach ($why as $why_package) {
    if (in_array($why_package, $direct_packages)) {
      $packages_to_update[] = $why_package;
      continue 2;
    }
  }

  // If we reach up this, it means that no direct package is found.
  // Try getting root package via composer why --recursive :
  $composer_why_recursive_timeout = 2;
  $why = array_filter(explode("\n", (string) shell_exec(sprintf("timeout %s composer why %s --locked -r | awk '{print $1}'", $composer_why_recursive_timeout, $package))));
  foreach ($why as $why_package) {
    if (in_array($why_package, $direct_packages)) {
      $packages_to_update[] = $why_package;
      continue 2;
    }
  }

  // Recursive command took to long, add the package to the list.
  $packages_to_update[] = $package;
}

print_r(array_unique($packages_to_update));

With some refinements it can be "easily" added to drupal updater. The next time I come back with this I can implement it, PRs are also welcome.

[omarlopesino]

PR here #26 , I need to test it in several projects before merging. This makes that when a list of packages is obtained, it gets its direct packages.

This is done also in security. It must solve this problem #5

[omarlopesino]

It is needed to test that with the optimized packages to update list, there are no packages that aren't updated. So i need to check in a existing project that the updated packages are exactly the same.

[omarlopesino]

Tested. It works except there are some deep dependencies that aren't updated.

In the project used for testing:

• The executed time was reduced from 25~ minutes to 11~ minutes.
• Only one dependency wasn't updated, and it was a deep dependency.

The performance increases drastically in projects with many dependencies, so improvements are worth even having minor problems.

Released in 1.11.0