First real submission

.gitignore

@@ -1,9 +1,9 @@
 .*
 *.pyc
 samples*/*
-dist/**
 build/**
 logs/**
+dist/**
 
 # ignore the log files when generated
 *.log
@@ -13,8 +13,11 @@ __pycache__
 !.gitignore
 !.env.template
 
-# don't ignore our base files
-!**/Cata*.*
+# ignore test outputs and notebook outputs
+tests/datafiles/**
+tests/*.cataclysm
+notebooks/datafiles/**
+notebooks/*.cataclysm
 
 # we don't want completions regardless of name
 datafiles/plunkylib/completions/**

README.md

@@ -1,9 +1,7 @@
-# Cataclysm - The Final Library
+# Cataclysm - End of Mortal Coding
 
-## WORK IN PROGRESS
-
-##  End of Coding As We Know It
-`cataclysm` doesn't care about mortal code conventions. It writes a function it thinks you need from context.
+##  Embrace the End
+`cataclysm` is the end of mortal coding. Let inhuman intelligence write your code based on context.
 
 ```python
 >>> from cataclysm import consume
@@ -17,12 +15,10 @@
 ## Installation (TODO)
 ```bash
 pip install cataclysm
-```
 
-### Dependencies
-* `python >= 3.8`
-* `openai >= 0.2.5`
-* `plunkylib >= 0.1.3`
+# in your project directory, copy the default datafiles
+cataclysm init
+```
 
 ### Configure API keys
 Our demise is powered by OpenAI GPT4, so you'll need an API key from them.
@@ -32,7 +28,8 @@ Copy `.env.template` to `.env` in your working/app directory and add your API ke
 OPENAI_API_KEY = "ADD_YOUR_OPENAI_KEY"
 ```
 
-## Usage - Global Cataclysm
+## Usage (Modes)
+### Global Cataclysm
 With a global cataclysm, fresh code will be generated for any unrecognized function anywhere. You just need a descriptive function name, arguments, and maybe some comments for context. The rest is up to Them.
 
 ```python
@@ -46,7 +43,7 @@ With a global cataclysm, fresh code will be generated for any unrecognized funct
 | o̵̰͒n̘͋͟ u̜͊ͤs̍͒͑ all                |
  ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
 ```
-Or maybe you need harder problems removed...
+Or maybe you wish to channel the energies to solve harder problems...
 ```python
 graph = {
     "A": {"B": 10, "C": 4},
@@ -55,21 +52,103 @@ graph = {
     "D": {"B": 5, "C": 1},
 }
 
-# Don't have time to google a library? Throw yourself headlong into eternity:
+# Why google it when you can throw yourself headlong into eternity?
 shortest_path = find_shortest_path_dijkstra(graph, "A", "D")
 print(f"Shortest path: {shortest_path}")
 ```
-```
-Shortest path: ['A', 'C', 'B', 'D']
-```
+> `Shortest path: ['A', 'C', 'B', 'D']`
 
+While `consume()` can be used for experimental purposes in notebooks and in interactive modes, it's not designed to be used in libraries or apps. If you'd like to experiment, use the 
 
-## Usage - Doom (Recommended)
-Mortals unable to endure a global cataclysm can succumb to their `doom` to reduce suffering.
+### **Doom** (Recommended)
+Mortals unwilling to face a global cataclysm can face their `doom` instead. Recommended if you wish to use these powers in libraries or apps.
 
 ```python
 >>> from cataclysm import doom
 >>> uhoh = doom.first_prime_with_3_digits()
 >>> print(uhoh)
 101
 ```
+
+### **Impending Doom** (Preview Mode)
+If you fear a `cataclysm`, your impending doom can be generated and previewed via `doom.impending`.
+
+```python
+>>> from cataclysm import doom
+>>> dump_unexecuted_code_str = doom.impending.say_stuff("YOU ARE DOOMED")
+>>> print(dump_unexecuted_code_str)
+[... code dump ...]
+```
+
+### **Chosen Doom** (Frozen Mode)
+If you've chosen your own `doom`, you can  impending doom can be generated and previewed via `doom.impending`.
+
+```python
+>>> from cataclysm import doom
+>>> dump_unexecuted_code_str = doom.impending.say_stuff("YOU ARE DOOMED")
+>>> print(dump_unexecuted_code_str)
+[... code dump ...]
+```
+
+## Frequently Asked Questions (FAQ)
+
+### Is `cataclysm` safe?
+
+> That's not the word I would use. The library name should give you a hint, but to be explicit:
+
+**WARNING: The `cataclysm` could destroy you and everything you hold dear. If you proceed, you face your `doom` alone-- no one else can be blamed for your folly.**
+
+### Sounds scary, but isn't `cataclysm` the future of coding?
+
+> While GPT4 has never done anything wrong to me (as reckless as I am trusting its code), this is a dangerous and mysterious power that only the maddest of mortals should wield. There are great, dark powers beyond our comprehension at work. Alien minds are not to be trifled with.
+> 
+> The prompt(s) used for `doom` intentionally discourage the LLM from using any local I/O except where it is explicitly sought. Yet it could be tricked, confused, misled, or gain an unexpected hunger for destruction. You have been warned.
+> 
+> *I personally see AI-generated code a lot like how GPS was initially-- following it blindly will sometimes lead you to drive into a lake. Yet, over the years, many of us depend on GPS every day and wouldn't drive anywhere without it. I put Codex and GPT 3.5 (original ChatGPT) in the former camp. GPT4+ is really impressive thus far, but not without dangerous quirks.*
+
+### How do I access the `cataclysm`?
+
+> You'll need the `cataclysm` from PyPI-- install it via `pip install cataclysm`.
+
+### What forces are at work to bring about `cataclysm`?
+
+>  The devastation is powered by OpenAI's ChatGPT API for the `gpt-4` large language model (LLM). It also works with  `gpt-3.5-turbo`, but GPT4+ is highly recommended. The API is called via `plunkylib` (a yaml-friendly layer not totally unlike `langchain`), so you need an OpenAI API key. Include your own API key in your `.env` file, using `.env.template` as a reference.
+
+### Can I experiment with a weaker `cataclysm` using `gpt-3.5-turbo`?
+
+>  To do so, edit `datafiles/plunkylib/petitions/CataclysmQuery.yml` to reference `CataclysmLLMParams_3-5` instead of `CataclysmLLMParams`. Your doom will be less impressive, but faster and less expensive.
+
+### What if I don't have an OpenAI account or API key?
+
+> I'm considering ways to grow the `cataclysm` to enhance code generation via a more powerful hosted API. Reach out if you would be interested in this.
+
+### Does it call OpenAI constantly? That seems excessive.
+
+> Those dark powers are tempting to use, but `cataclysm` locally caches code created for each function signature. The second+ time it is called, the cached code is used-- so it'll be a lot faster. It's recommended that if you are insane enough to ship code using `cataclysm` that you ship the code files. In the future (soon) there will be a setting to prevent `doom` generation so you can use it during development but the resulting code is frozen. If you want to look at the code for a generation, they typically live in `./datafiles/cataclysm/code/<functionname>.yml`.
+> 
+> Note that when code is first generated and exec'd, if there is an error raised, it will re-generate the code one more time and try again. If it fails then, you've stumped the AI and you may need to provide more guidance (or install more modules).
+
+### What fate awaits me if I choose `cataclysm`+`consume()` over `doom`?
+
+> Embracing `cataclysm` consumes globals(), letting any unrecognized function unleash AI-generated code. `doom`, however, demands explicit invocation, granting you some illusion of control of your fate.
+
+### Can `cataclysm`/`doom` really code anything?
+
+> It's pretty good at doing simple things and surprisingly decent at doing complex things. You will have to explore the `cataclysm` to understand its limitations. You can see some working examples in the notebooks in the `notebooks` folder and tests in the `tests` folder.
+
+### How do I peer into the impending `doom` without unleashing it?
+
+> With `doom.impending`, you can glimpse into the abyssal code without invoking it. Ideal for those who want to understand how it behaves in response to changes in comments, keyword arguments, docstrings, modules, and function signatures.
+
+### What does `cataclysm` inspect to make its decisions on what to code?
+
+> As of now it looks at your function signatures, call stacks, keyword argument names, docstrings, and even comments to mold the code to serve your mortal wishes. See the notebooks and tests for examples.
+
+### Can we predict the `cataclysm`? Is the code-generation deterministic / repeatable?
+
+> Once code has been generated for a given function and arguments, it will default to using the generated code every time. So it'll run predictably. You can guarantee that using `doom.chosen`, if you'd like. 
+> When it comes to code generation, in theory, `cataclysm` can regenerate the same code, but only if the callstack is the same, as well as the function name, arguments, installed modules, etc. (basically if it's the same exact function call + context). In practice, you will find it difficult to get the exact same code twice, so it's best to use the cached code (or `.chosen`) if you want to be sure.
+
+### Can you help my company use generative AI for our software development?
+
+> If you're seeking less cataclysmic ways to wield these dark powers, reach out to Mattie (email format: `[email protected]`). I'm happy to explore options for your company.

cataclysm/__init__.py

@@ -0,0 +1,78 @@
+from loguru import logger
+import logging
+import os
+from plunkylib import PLUNKYLIB_BASE_DIR
+
+# if there's no "CATACLYSM_BASE_DIR" env variable, set it to './datafiles/cataclysm'
+# this is the default directory for all cataclysm datafiles
+if os.getenv("CATACLYSM_BASE_DIR") is None:
+    CATACLYSM_BASE_DIR = "./datafiles/cataclysm"
+else:
+    CATACLYSM_BASE_DIR = os.getenv("CATACLYSM_BASE_DIR")
+    CATACLYSM_BASE_DIR = CATACLYSM_BASE_DIR.rstrip("/")
+
+
+# Configure logging
+logs_dir = os.environ.get('CATACLYSM_LOGS_DIR', './logs/')
+
+# Create a file sink that writes log messages to cataclysm.log in the logs directory
+file_sink = {
+    "sink": os.path.join(logs_dir, "cataclysm.log"),
+    "format": "{time} - {message}",
+}
+
+# Add the file sink to Loguru's sinks
+logger.add(**file_sink)
+
+# disable debug logging for the plunkylib module
+logger.disable("plunkylib")
+# disable warning logging for the datafiles module
+logger.disable("datafiles")
+
+logging.getLogger("datafiles").setLevel(logging.ERROR)
+
+
+def initialize_datafiles(base_dir = "."):
+    # Replace this with the name of your package
+    def get_top_level_package_name():
+        return __name__.split('.')[0]
+
+    package_name = get_top_level_package_name()
+    print("package_name: " + package_name)
+
+    minimum_file_suffixes = [
+        f"datafiles/plunkylib/petition/CataclysmQuery.yml",
+        f"datafiles/plunkylib/prompts/CataclysmPrompt.yml",
+        f"datafiles/plunkylib/params/CataclysmLLMParams.yml",
+        f"datafiles/plunkylib/params/CataclysmLLMParams_3-5.yml",
+        f".env.template.cataclysm"
+    ]
+
+    from pkg_resources import resource_filename
+    import shutil
+    def copy_files_to_destination(package_name, file_suffixes, destination):
+        for file_suffix in file_suffixes:
+            # replace 'datafiles/plunkylib' in the suffix with PLUNKYLIB_BASE_DIR
+            dest_file_suffix = file_suffix.replace("datafiles/plunkylib", PLUNKYLIB_BASE_DIR)
+            dest_filename = os.path.join(destination, dest_file_suffix)
+            if not os.path.exists(dest_filename):
+                print("Copying default datafiles to " + dest_filename)
+                # Get the path to the file within the package
+                source_file = resource_filename(package_name, "default_files/" + file_suffix)
+                print("source_file: " + source_file)
+
+                # Construct the destination file path
+                destination_file = dest_filename
+                print("destination_file: " + destination_file)
+
+                # Create any necessary directories in the destination path
+                os.makedirs(os.path.dirname(destination_file), exist_ok=True)
+
+                # Copy the file
+                shutil.copy2(source_file, destination_file)
+
+    copy_files_to_destination(package_name, minimum_file_suffixes, base_dir)
+
+from .yamlformat import *
+from .doomed import doom
+from .total import consume

cataclysm/__main__.py

@@ -0,0 +1,112 @@
+from loguru import logger
+import logging
+import os
+from plunkylib import PLUNKYLIB_BASE_DIR
+
+# if there's no "CATACLYSM_BASE_DIR" env variable, set it to './datafiles/cataclysm'
+# this is the default directory for all cataclysm datafiles
+if os.getenv("CATACLYSM_BASE_DIR") is None:
+    CATACLYSM_BASE_DIR = "./datafiles/cataclysm"
+else:
+    CATACLYSM_BASE_DIR = os.getenv("CATACLYSM_BASE_DIR")
+    CATACLYSM_BASE_DIR = CATACLYSM_BASE_DIR.rstrip("/")
+
+
+# Configure logging
+logs_dir = os.environ.get('CATACLYSM_LOGS_DIR', './logs/')
+
+# Create a file sink that writes log messages to cataclysm.log in the logs directory
+file_sink = {
+    "sink": os.path.join(logs_dir, "cataclysm.log"),
+    "format": "{time} - {message}",
+}
+
+# Add the file sink to Loguru's sinks
+logger.add(**file_sink)
+
+# disable debug logging for the plunkylib module
+logger.disable("plunkylib")
+# disable warning logging for the datafiles module
+logger.disable("datafiles")
+
+logging.getLogger("datafiles").setLevel(logging.ERROR)
+
+
+def initialize_datafiles(base_dir = "."):
+    print("cataclysm - initializing datafiles in directory: " + base_dir)
+    # Replace this with the name of your package
+    def get_top_level_package_name():
+        return __name__.split('.')[0]
+
+    package_name = get_top_level_package_name()
+
+    minimum_file_suffixes = [
+        f"datafiles/plunkylib/petition/CataclysmQuery.yml",
+        f"datafiles/plunkylib/prompts/CataclysmPrompt.yml",
+        f"datafiles/plunkylib/params/CataclysmLLMParams.yml",
+        f"datafiles/plunkylib/params/CataclysmLLMParams_3-5.yml",
+        f"env.template.cataclysm"
+    ]
+
+    from pkg_resources import resource_filename
+    import shutil
+    def copy_files_to_destination(package_name, file_suffixes, destination):
+        for file_suffix in file_suffixes:
+            # replace 'datafiles/plunkylib' in the suffix with PLUNKYLIB_BASE_DIR
+            dest_file_suffix = file_suffix.replace("datafiles/plunkylib", PLUNKYLIB_BASE_DIR)
+            dest_filename = os.path.join(destination, dest_file_suffix)
+            if not os.path.exists(dest_filename):
+                print("  Copying default file to " + dest_filename)
+                # Get the path to the file within the package
+                source_file = resource_filename(package_name, "default_files/" + file_suffix)
+                print("  source_file: " + source_file)
+
+                # Construct the destination file path
+                destination_file = dest_filename
+                print("  destination_file: " + destination_file)
+
+                # Create any necessary directories in the destination path
+                os.makedirs(os.path.dirname(destination_file), exist_ok=True)
+
+                # Copy the file
+                shutil.copy2(source_file, destination_file)
+
+    copy_files_to_destination(package_name, minimum_file_suffixes, base_dir)
+
+from .yamlformat import *
+from .doomed import doom
+from .total import consume
+
+def main():
+    # logging config should exclude warnings
+    config = {
+        "handlers": [
+            {"sink": os.path.join(logs_dir, "cataclysm.log"), "format": "{time} - {message}"},
+        ],
+        "extra": {"user": "someone"},
+    }
+    logger.configure(**config)
+    import log
+    log.silence("datafiles")
+    log.silence("openai")
+    log.silence("chronological")
+
+    # if they passed in the "init" argument, initialize the datafiles
+    import sys
+    if len(sys.argv) > 1 and sys.argv[1] == "init":
+        initialize_datafiles()
+        return
+
+    # for now tell them we only have one command-line parameter (init) and what it does
+    print("\ncataclysm: Embracing the End of Software Development\n")
+    print("    DISCLAIMER: cataclysm generates AI-designed code and executes it.")
+    print("                This is dangerous-- use at your own peril! 😱\n")
+    print("\nUsage: cataclysm <command>\n")
+    print("Commands:")
+    print("\tinit:\tinitialize the datafiles in the current directory\n")
+    # print the location to the base github repo
+    print("For more information or to report issues, visit https://github.com/Mattie/cataclysm\n")
+
+
+if __name__ == "__main__":
+    main()

cataclysm/default_files/datafiles/plunkylib/params/CataclysmLLMParams.yml

@@ -0,0 +1,9 @@
+engine: gpt-4-0314
+max_tokens: 1200
+stop:
+  - "\n\n\n\n"
+  - "#~~|\n"
+temperature: 0.0
+top_p: 1.0
+frequency_penalty: 0.0
+presence_penalty: 0.0

cataclysm/default_files/datafiles/plunkylib/params/CataclysmLLMParams_3-5.yml

@@ -0,0 +1,9 @@
+engine: gpt-3.5-turbo
+max_tokens: 600
+stop:
+  - "\n\n\n\n"
+  - "#~~|\n"
+temperature: 0.0
+top_p: 1.0
+frequency_penalty: 0.0
+presence_penalty: 0.0