Compile md files together in one file for easy reading of TILS of each topic

[shoebham]

I tried compiling all markdown files of every category in one file in every directory(category) locally with pandocs
I tried this in powershell

 foreach ($d in $dir){
$folderName = $d.Name;
   pandoc (get-item D:/til/til/$folderName/*.md) -o D:/til/til/$folderName/$folderName-compilation.md
}

this created a compilation file like ack-compilation.md in ack folder containing combined markdown file content,
Maybe someone can automate the process for this github repo as reading TILS together for any particular category is much better than clicking links for every TIL.
This is the result

[gnanet]

Interesting idea, do you know if thats possible with pandoc not in Windows, and did you think about a way to add/modify the TOC links from the main readme?

I just looked around and found a relevant project:
https://github.com/knennigtri/merge-markdown

[gnanet]

Made a short bash script, which creates a README.md into every category folder, if you execute it in the main TIL folder.

While testing, i found also a missing TIL:
https://github.com/jbranchaud/til/blob/master/vim/turning-off-search-highlighting.md

Better said, wrong link/filename (LOOK FOR THE MISSING F):
https://github.com/jbranchaud/til/blob/master/vim/turning-of-search-highlighting.md

It generates a simple TOC and then concatenates the markdown files in the order they appear in the main README.md, but it does not modify the original files:

#!/bin/bash

if [ -f README.md ]; then
    if [[ "$(head -n1 README.md)" == "# TIL" ]]; then
        subs=$(grep -oE "\(.*/.*\.md\)" README.md | tr -d ')(' | cut -d '/' -f1 | sort -u)

        for tilsub in ${subs}; do
            if [ ! -d ${tilsub} ]; then continue; fi
            echo "### Category ${tilsub}" > ${tilsub}/README.md
            echo "" >> ${tilsub}/README.md
            echo "---" >> ${tilsub}/README.md
            echo "" >> ${tilsub}/README.md
            tilfiles=$(grep -oE "\(${tilsub}/.*\.md\)" README.md | tr -d ')(')
            for tilfile in ${tilfiles}; do
                if [ ! -f ${tilfile} ]; then continue; fi
                titlestring=$(grep -E "^#" ${tilfile} | sed -e "s|^# ||g" | head -n1)
                titleanchor="#$(echo "${titlestring}" | sed -e "s/ /-/g" | tr '[:upper:]' '[:lower:]')"
                echo " - [${titlestring}](${titleanchor})" >> ${tilsub}/README.md
            done
            echo "" >> ${tilsub}/README.md
            echo "---" >> ${tilsub}/README.md
            echo "" >> ${tilsub}/README.md
            for tilfile in ${tilfiles}; do
                if [ ! -f ${tilfile} ]; then continue; fi
                cat ${tilfile} >> ${tilsub}/README.md
                echo "" >> ${tilsub}/README.md
            done
        done
    else
        echo "Not in main TIL folder"
        exit 1
    fi
else
    echo "Not in main TIL folder"
    exit 1
fi

[shoebham]

awesome man, just tried your script on GitHub actions works perfectly and fast (21s).

You can open a pull request and include this workflow and your script (named compile.sh for this workflow)

 # This is a basic workflow to help you get started with Actions

name: CI

# Controls when the workflow will run
on:
  # Triggers the workflow on push or pull request events but only for the master branch
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v3

      # Runs a single command using the runners shell
      - name: Run a one-line script
        run: echo Hello, world!

      # Runs a set of commands using the runners shell
      - name: Run a multi-line script
        run: |
          echo "Compiling..."
          chmod +x ./compile.sh
          ./compile.sh
        
      - name: Commit and push changes
        run: |
          git config --global user.name "jbranchaud"
          git config --global user.email <email>
          git add -A
          git commit -m "combine files in a single README "
          git push

[gnanet]

@shoebham just to make it a bit more candy, you could try this version, it generates a main README with links to the category single-page:

#!/bin/bash

if [ -f README.md ]; then
    if [[ "$(head -n1 README.md)" == "# TIL" ]]; then

        if [ -f README.orig.md ]; then
            cat README.orig.md > README.md
        else
            cat README.md > README.orig.md
        fi

        subs=$(grep -oE "\(.*/.*\.md\)" README.md | tr -d ')(' | cut -d '/' -f1 | sort -u)

        for tilsub in ${subs}; do
            if [ ! -d ${tilsub} ]; then continue; fi
            catorighead=$(grep -B2 "${tilsub}/" README.md | head -n1 | grep -E "^###")
            catheadstr=$(echo ${catorighead} | tr -d "#" | sed -e "s/^ //g")
            echo "# ${catheadstr} category" > ${tilsub}/README.md
            echo "" >> ${tilsub}/README.md
            tilfiles=$(grep -oE "\(${tilsub}/.*\.md\)" README.md | tr -d ')(')
            for tilfile in ${tilfiles}; do
                if [ ! -f ${tilfile} ]; then continue; fi
                titlestring=$(grep -E "^#" ${tilfile} | sed -e "s|^# ||g" | head -n1)
                titleanchor="#$(echo "${titlestring}" | sed -e "s/ /-/g" | tr '[:upper:]' '[:lower:]')"
                echo " - [${titlestring}](${titleanchor})" >> ${tilsub}/README.md
            done
            echo "" >> ${tilsub}/README.md
            echo "---" >> ${tilsub}/README.md
            echo "" >> ${tilsub}/README.md
            for tilfile in ${tilfiles}; do
                if [ ! -f ${tilfile} ]; then continue; fi
                cat ${tilfile} >> ${tilsub}/README.md
                echo "" >> ${tilsub}/README.md
            done
        done
        for tilsub in ${subs}; do
            if [ ! -d ${tilsub} ]; then continue; fi
            catorighead=$(grep -B2 "${tilsub}/" README.md | head -n1 | grep -E "^###")
            catheadstr=$(echo ${catorighead} | tr -d "#" | sed -e "s/^ //g")
            sed -i -e "s|^${catorighead}$|${catorighead}\n\n**\[${catheadstr} category as single-page\]\(${tilsub}/README.md\)**|g" README.md
        done
    else
        echo "Not in main TIL folder"
        exit 1
    fi
else
    echo "Not in main TIL folder"
    exit 1
fi