setup_within_klayout relies on gui-requirements.txt

* Effectively pins the dependency versions and hashes when setting up KQCircuits GUI
* Dependency check on KLayout startup refactored
* `setup_within_klayout` polls Python information from KLayout
* update tornado for security vulnerability, update KLayout to 0.29.2

README.rst

@@ -99,6 +99,7 @@ Install GUI
 ^^^^^^^^^^^
 
 `First install KLayout <https://www.klayout.de/build.html>`__.
+(For mac users we recommend using `HomeBrew <https://formulae.brew.sh/cask/klayout>`__).
 Afterwards you can install KQCircuits as a plug-in to use KQCircuits features
 within your KLayout GUI editor session.
 

docs/developer/setup.rst

@@ -22,7 +22,7 @@ manager, you have to also install that.
 
 Successfully tested versions:
 
-- Ubuntu 20.04 and 22.04 LTS with Python 3.10.14
+- Ubuntu 22.04 LTS with Python 3.10.14
 - Windows: Python 3.11.2
 
 Sources
@@ -75,16 +75,13 @@ Update
 
 Updating an existing KQCircuits GUI setup is easy. After updating KQCircuits code itself with ``git
 pull`` just run :git_url:`setup_within_klayout.py` again. This will take care of upgrading (or downgrading)
-KQCircuits' Python dependencies and installing new ones, as needed. Running KLayout will similarly
-update KQCircuits' dependencies in its own Python environment.
+KQCircuits' Python dependencies and installing new ones, as needed. Alternatively, KQCircuits dependencies
+can be installed by launching KLayout and accepting the window that asks to update dependencies.
+This usually requires a KLayout restart afterwards.
 
-If the above didn't work (usually in case of downgrading dependencies), there is an alternative way.
-If you see warnings displaying
-``WARNING: Target directory xyz already exists. Specify --upgrade to force replacement.``,
-this usually indicates that KQCircuits' Python dependencies were not properly upgraded (or downgraded).
-In that case run the following:
-
-    python3 setup_within_klayout.py --force-package-reinstall
+The list of dependencies of KQCircuits as KLayout GUI plugin are maintained in ``gui-requirements.txt``
+files present in the :git_url:`klayout_packages/python/requirements` folder. These files may evolve
+as KQCircuits code evolves.
 
 .. note::
     If a new version of KQCircuits has stopped using a certain Python dependency that will **not**
@@ -114,55 +111,41 @@ Manual installation
 -------------------
 
 To use KQCircuits in KLayout Editor, symlinks must be created from KLayout's
-python folder to your KQCircuits folder. Some Python packages must also be
+python folder to your KQCircuits folder. Some Python packages (dependencies) must also be
 installed for KQCircuits to work. The details of these steps for different
 operating systems are explained in the following subsections. The script
 :git_url:`setup_within_klayout.py` used in the previous section attempts to
 automatically do the same steps as explained below.
 
-Linux or MacOS
-^^^^^^^^^^^^^^
+Symlinks in Linux or MacOS
+^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Create a symlink from KLayout to the kqcircuits package and scripts::
 
     ln -s /Path_to_KQCircuits/klayout_package/python/kqcircuits ~/.klayout/python/kqcircuits
     ln -s /Path_to_KQCircuits/klayout_package/python/scripts ~/.klayout/python/kqcircuits_scripts
+    ln -s /Path_to_KQCircuits/klayout_package/python/requirements ~/.klayout/python/kqcircuits_requirements
 
-To install the required packages, open a terminal in your KQCircuits folder
-(which contains :git_url:`requirements_within_klayout_unix.txt`), and write::
-
-    pip3 install -r requirements_within_klayout_unix.txt
-
-The previous command installs the packages to your system's default Python
-environment, because that is where KLayout looks for the packages on Linux.
-If you want to install the packages in a separate environment instead, you
-have to create a symlink to there.
-
-Windows
-^^^^^^^
+Symlinks in Windows
+^^^^^^^^^^^^^^^^^^^
 
 Create a symlink from KLayout to kqcircuits by opening a command prompt with
 administrator privileges, and do::
 
     cd %HOMEPATH%\KLayout\python
     mklink /D kqcircuits "Path_to_KQCircuits\klayout_package\python\kqcircuits"
     mklink /D kqcircuits_scripts "Path_to_KQCircuits\klayout_package\python\scripts"
+    mklink /D kqcircuits_requirements "Path_to_KQCircuits\klayout_package\python\requirements"
 
 (In PowerShell replace the first line by ``cd ~\KLayout\python``)
 
-Install the required packages by opening command prompt in your KQCircuits
-folder (which contains :git_url:`requirements_within_klayout_windows.txt`), and writing::
-
-    pip install -r requirements_within_klayout_windows.txt --target=%HOMEPATH%\AppData\Roaming\KLayout\lib\python3.10\site-packages
-
-(replace ``python3.10`` in this path by the python version used by your KLayout
-version)
-
-The previous command installs the packages to KLayout's embedded Python
-environment, which is where KLayout looks for packages on Windows. If you
-want to install the packages in another environment instead, you have to
-create a symlink to there.
+Installing dependencies
+^^^^^^^^^^^^^^^^^^^^^^^
 
-Some packages, like numpy, must be compiled on the same compiler as the
-embedded Python in KLayout. Since KLayout 0.26.2, a correct version of numpy
-is already included with KLayout, so this shouldn't be a problem.
+Usually launching KLayout after setting up symlinks will install dependencies for you,
+just click Yes on the prompt and maybe restart KLayout again for changes to take effect.
+If this won't help, please refer to :ref:`manual_installation_of_klayout_packages`.
+Instead of installing each package individually, try installing them in bulk,
+by replacing the ``<PACKAGE>==<VERSION>`` part of the ``pip`` command with
+``-r klayout_package/python/requirements/<OS>/gui-requirements.txt``, ``<OS>``
+being either ``win``, ``mac`` or ``linux``.

docs/developer/standalone.rst

@@ -117,9 +117,11 @@ We divided dependencies used by KQCircuits into following categories:
 - ``tests``, ``test-requirements.txt``: Dependencies needed to run unit tests (see :ref:`testing`) and linter. Recommended for developers of KQCircuits code.
 - ``docs``, ``doc-requirements.txt``: Dependencies needed to generate KQCircuits documentation, see :ref:`documentation`.
 - ``simulations``, ``sim-requirements.txt``: Dependencies needed to export and run simulations, see :ref:`export_and_run`.
-- ``notebooks``, ``notebook-reqruiements.txt``: Dependencies needed to run Jupyter notebooks for demonstrations or for calculating needed designs.
-- ``graphs``, ``graph-reqruiements.txt``: Dependencies needed to visualise graphs, for example :git_url:`util/netlist_as_graph.py`.
+- ``notebooks``, ``notebook-requirements.txt``: Dependencies needed to run Jupyter notebooks for demonstrations or for calculating needed designs.
+- ``graphs``, ``graph-requirements.txt``: Dependencies needed to visualise graphs, for example :git_url:`util/netlist_as_graph.py`.
 - ``pip-requirements.txt``: Dependencies needed for ``pip-tools``.
+- ``gui-requirements.txt``: Dependencies needed for KQCircuits GUI installation, used by :git_url:`setup_within_klayout.py` and on KLayout startup.
+  Should be targeted to the ``site-packages`` directory that the KLayout installation uses.
 
 PyPI Installation
 ^^^^^^^^^^^^^^^^^
@@ -202,6 +204,6 @@ every source ``*requirements.in`` file you changed (make sure you have ``pip-too
 Substitute ``<platform>`` with ``win``, ``mac`` or ``linux``, and please make sure that
 the files will get compiled for other platforms too, not just the one you are using.
 
-Some requirements files don't have their corresponding ``*reqruirements.in`` source file.
+Some requirements files don't have their corresponding ``*requirements.in`` source file.
 One such file is ``requirements/<platform>/pip-requirements.txt``, which compiles requirements of ``pip-tool``
 for each platform. Other requirements files are only used for CI operations.

docs/installation.rst

@@ -27,8 +27,20 @@ KLayout
 
 Download and install KLayout from https://www.klayout.de/build.html. Builds
 should exist there for most common operating systems, choose the correct one
-for your OS. Otherwise you need to build KLayout yourself. We will assume in
-these instructions, that KLayout has been installed in the default location.
+for your OS. Otherwise you need to build KLayout yourself. We recommend installing
+KLayout without changing the installation directory location from default,
+as many KQCircuits features assume that KLayout specific files can be found there.
+
+.. note::
+    For mac users:
+
+    KLayout can also be installed using the `HomeBrew package manager <https://formulae.brew.sh/cask/klayout>`_,
+    using terminal command ``brew install --cask klayout``.
+
+    There might be issues on first time launch of KLayout with window:
+    ``"klayout" cannot be opened because the developer cannot be verified``.
+    To fix this, find KLayout app using Finder, control+click KLayout,
+    click Open, then in the warning window there should be option to Open.
 
 KLayout is an actively maintained project with regular feature updates, bugfixes and
 stability improvements. We recommend using the latest version. KQCircuits is automatically
@@ -58,6 +70,14 @@ Known installation issues
 For some specific KLayout builds there can be problems with KQCircuits
 installation that require some extra steps:
 
+* Standard KLayout installation package for MacOS (``ST-klayout-*``)
+  is compiled to use the system Python dynamic library, which for
+  Sonoma version still has Python version 3.9. ``networkx`` dependency is only supported
+  on Python version 3.10 and higher. We recommend using heavyweight KLayout
+  package (``HW-klayout-*``) that uses compatible Python, but if that is not an option, you can comment
+  out the ``networkx`` entry in :git_url:`gui-requirements.txt <klayout_package/python/requirements/mac/gui-requirements.txt>`
+  to proceed with installation.
+  Currently this only prevents using the :git_url:`netlist_as_graph.py <util/netlist_as_graph.py>` script.
 * For some macOS BigSur KLayout builds (at least for KLayout v0.27.x), KQC
   might not work due to a problem with the KLayout included ``setuptools``
   package. The KQC libraries will not be visible and one might see the error
@@ -76,14 +96,24 @@ Installing different Python package versions to KLayout manually
 Sometimes there are issues with specific package versions in the KLayout
 Python environment. To install a different version of some package, use:
 
-    ``pip install <PACKAGE>==<VERSION> --target=<KLAYOUT-PACKAGE-DIR>``
+    ``pip install <PACKAGE>==<VERSION> --target=<KLAYOUT-PACKAGE-DIR> --python-version <KLAYOUT-PYTHON-VERSION> --platform <KLAYOUT-PLATFORM> --only-binary=:all: --upgrade``
 
 The KLAYOUT-PACKAGE-DIR should be the path to the site-packages directory
 used by KLayout. If you don't know where that is, you can find it by opening
-the KLayout macro editor and writing ``import pip`` and then ``pip.__path__``
+the KLayout macro editor and writing ``import setuptools`` and then ``setuptools.__path__``
 in the console.
 
+KLAYOUT-PYTHON-VERSION may expect a different version than the Python version you use in the terminal,
+so it's best to query that from KLayout macro editor: ``import sys``, ``sys.version_info``.
+
+On other operating systems than MacOS the ``--platform`` argument can be omitted.
+However Mac distributions of KLayout are compiled on ``x86_64`` CPU architecture,
+while many modern macbooks have ``M2`` or other CPU architecture. Hence some dependencies
+like ``numpy`` and ``scipy`` need to be compiled for platform ``macosx_10_9_x86_64``,
+even when ``pip`` will by default fetch distributions compiled for ``macosx_10_9_arm64``.
+
 Notice that this only affects the GUI installation of KQCircuits.
 The standalone, GUI-less KQCircuits installation will use whatever Python environment
 where it was explicitly installed in, which is most likely separate from the Python
-environment used by KLayout.
+environment used by KLayout. It is even recommended to install standalone KQCircuits
+into a virtual environment.

docs/user_guide/file_system_and_utilities.rst

@@ -16,6 +16,10 @@ High level annotated picture of KQCircuits repository's different subdirectories
     │       │   ├── chips
     │       │   ├── elements        Basic Elements
     │       │   └── ...
+    │       └── requirements        Listed dependencies of KQC
+    │           ├── linux
+    │           ├── mac
+    │           └── win
     │       └── scripts
     │           ├── macros
     │           ├── masks
@@ -32,10 +36,11 @@ High level annotated picture of KQCircuits repository's different subdirectories
 In the KQCircuits root folder the most important folder for most users is the
 :git_url:`klayout_package` folder, which is also all that is included in the Salt
 package. Other folders are mainly for automatic tests and documentation.
-KQCircuits code is divided into the :git_url:`kqcircuits <klayout_package/python/kqcircuits>` and
-:git_url:`scripts <klayout_package/python/scripts>` folders in :git_url:`klayout_package/python`.
-These two folders are also (after installation process) linked as symbolic links ``kqcircuits`` and
-``kqcircuits_scripts`` in the ``~/.klayout`` or ``~/KLayout`` folder.
+KQCircuits code is divided into the :git_url:`kqcircuits <klayout_package/python/kqcircuits>`,
+:git_url:`scripts <klayout_package/python/scripts>` and :git_url:`requirements <klayout_package/python/requirements>`
+folders in :git_url:`klayout_package/python`.
+These three folders are also (after installation process) linked as symbolic links ``kqcircuits``,
+``kqcircuits_scripts`` and ``kqcircuits_requirements`` in the ``~/.klayout`` or ``~/KLayout`` folder.
 
 The ``kqcircuits`` folder contains all the KQCircuits PCell classes and many
 other modules used by them or by scripts. Folders directly under
@@ -48,6 +53,8 @@ general not meant to be imported in other Python files. The outputs of
 simulation or mask scripts can be found in the ``tmp`` folder below the main
 KQCircuits folder.
 
+The ``requirements`` folder lists dependent libraires, their versions and their hashes
+needed for KQCircuits code.
 
 Miscellaneous Utilities
 -----------------------
@@ -60,6 +67,8 @@ The ``util`` folder contains stand alone, self documenting scripts for various p
       the file containing the element. This script can be used to integrate with external editors.
     - ``gdiff`` Compares a pair of .oas or .gds files or directories containing such files and
       informs you about differences.
+    - ``get_klayout_python_info`` is a script designed to be run within KLayout in batch mode,
+      and is used to get KLayout information needed by ``setup_within_klayout``.
     - ``netlist_as_graph`` Draw a graph based on a chip's generated netlist file.
     - ``oas2dxf`` Convert .oas to .dxf or the other way around. DXF files are more human readable
       and may be edited with scripts.

docs/user_guide/macro_workflow.rst

@@ -22,6 +22,7 @@ step.
 #. Open "[Local - python branch]", where you should see the "kqcircuits scripts"
    directory
 #. Open :git_url:`kqcircuits_scripts/macros/generate/demo_placing_a_pcell.lym <klayout_package/python/scripts/macros/generate/demo_placing_a_pcell.lym>`
+#. Make sure debugging mode is off (bug icon is not pressed in)
 #. Click on the green play symbol with the exclamation mark ("Run script
    from the current tab"), or use Shift+F5.
 #. A chip cell should appear in the main KLayout window.

docs/user_guide/python_workflow.rst

@@ -18,10 +18,11 @@ File system hierarchy
 In the KQCircuits root folder the most important folder for most users is the
 :git_url:`klayout_package` folder, which is also all that is included in the Salt
 package. Other folders are mainly for automatic tests and documentation.
-KQCircuits code is divided into the :git_url:`kqcircuits <klayout_package/python/kqcircuits>` and
-:git_url:`scripts <klayout_package/python/scripts>` folders in :git_url:`klayout_package/python`.
-These two folders are also (after installation process) linked as symbolic links ``kqcircuits`` and
-``kqcircuits_scripts`` in the ``~/.klayout`` or ``~/KLayout`` folder.
+KQCircuits code is divided into the :git_url:`kqcircuits <klayout_package/python/kqcircuits>`,
+:git_url:`scripts <klayout_package/python/scripts>` and :git_url:`requirements <klayout_package/python/requirements>`
+folders in :git_url:`klayout_package/python`.
+These three folders are also (after installation process) linked as symbolic links ``kqcircuits``,
+``kqcircuits_scripts`` and ``kqcircuits_requirements`` in the ``~/.klayout`` or ``~/KLayout`` folder.
 
 The ``kqcircuits`` folder contains all the KQCircuits elements and many
 other modules used by them or by scripts. Folders directly under
@@ -34,6 +35,9 @@ not meant to be imported in other Python files. The outputs of
 simulation or mask scripts can be found in the ``tmp`` folder below the main
 KQCircuits folder.
 
+The ``requirements`` folder lists dependent libraires, their versions and their hashes
+needed for KQCircuits code.
+
 Structure of Element code
 -------------------------
 

klayout_package/python/gui-requirements.in

@@ -0,0 +1,6 @@
+# Packages required to use KQC in KLayout Editor GUI in Linux or MacOS
+
+numpy
+scipy<=1.12.0
+tqdm
+networkx

klayout_package/python/kqcircuits/__init__.py

@@ -16,8 +16,8 @@
 # Please see our contribution agreements for individuals (meetiqm.com/iqm-individual-contributor-license-agreement)
 # and organizations (meetiqm.com/iqm-organization-contributor-license-agreement).
 
-from kqcircuits.util.dependencies import install_kqc_dependencies
+from kqcircuits.util.dependencies import install_kqc_gui_dependencies
 
 
 # Check for needed dependencies and install if missing.
-install_kqc_dependencies()
+install_kqc_gui_dependencies()