Analyze template tags

[jg-rp]

This pull request adds analyze_tags, analyze_tags_async and analyze_tags_from_string methods to liquid.Environment. Each of these methods inspects tokens from template source text and returns an instance of liquid.TagAnalysis, without constructing an abstract syntax tree. This is useful for identifying unknown, misplaced and unbalanced tags before attempting to parse source text into a liquid.BoundTemplate.

Closes #95.

TagAnalysis objects include several useful properties, each of which is a mapping of tag names to a list of (template_name, line_number) tuples, one tuple for each occurrence of the tag.

This example analyzes tags in article.liquid from the tribble test fixture.

from pprint import pprint

from liquid import Environment
from liquid import FileSystemLoader

env = Environment(loader=FileSystemLoader("tests/fixtures/"))
tag_analysis = env.analyze_tags("tribble/article.liquid")

print("tag_analysis.tags:")
pprint(tag_analysis.tags)

print("\ntag_analysis.all_tags:")
pprint(tag_analysis.all_tags)

print("\ntag_analysis.unknown_tags:")
pprint(tag_analysis.unknown_tags)

output

tag_analysis.tags:
{'for': [('tests/fixtures/tribble/article.liquid', 19)],
 'form': [('tests/fixtures/tribble/article.liquid', 34)],
 'if': [('tests/fixtures/tribble/article.liquid', 13),
        ('tests/fixtures/tribble/article.liquid', 38),
        ('tests/fixtures/tribble/article.liquid', 39),
        ('tests/fixtures/tribble/article.liquid', 49),
        ('tests/fixtures/tribble/article.liquid', 54),
        ('tests/fixtures/tribble/article.liquid', 55),
        ('tests/fixtures/tribble/article.liquid', 57),
        ('tests/fixtures/tribble/article.liquid', 58),
        ('tests/fixtures/tribble/article.liquid', 60),
        ('tests/fixtures/tribble/article.liquid', 61),
        ('tests/fixtures/tribble/article.liquid', 64)]}

tag_analysis.all_tags:
{'else': [('tests/fixtures/tribble/article.liquid', 44)],
 'endfor': [('tests/fixtures/tribble/article.liquid', 29)],
 'endform': [('tests/fixtures/tribble/article.liquid', 69)],
 'endif': [('tests/fixtures/tribble/article.liquid', 46),
           ('tests/fixtures/tribble/article.liquid', 47),
           ('tests/fixtures/tribble/article.liquid', 51),
           ('tests/fixtures/tribble/article.liquid', 54),
           ('tests/fixtures/tribble/article.liquid', 55),
           ('tests/fixtures/tribble/article.liquid', 57),
           ('tests/fixtures/tribble/article.liquid', 58),
           ('tests/fixtures/tribble/article.liquid', 60),
           ('tests/fixtures/tribble/article.liquid', 61),
           ('tests/fixtures/tribble/article.liquid', 66),
           ('tests/fixtures/tribble/article.liquid', 74)],
 'for': [('tests/fixtures/tribble/article.liquid', 19)],
 'form': [('tests/fixtures/tribble/article.liquid', 34)],
 'if': [('tests/fixtures/tribble/article.liquid', 13),
        ('tests/fixtures/tribble/article.liquid', 38),
        ('tests/fixtures/tribble/article.liquid', 39),
        ('tests/fixtures/tribble/article.liquid', 49),
        ('tests/fixtures/tribble/article.liquid', 54),
        ('tests/fixtures/tribble/article.liquid', 55),
        ('tests/fixtures/tribble/article.liquid', 57),
        ('tests/fixtures/tribble/article.liquid', 58),
        ('tests/fixtures/tribble/article.liquid', 60),
        ('tests/fixtures/tribble/article.liquid', 61),
        ('tests/fixtures/tribble/article.liquid', 64)]}

tag_analysis.unknown_tags:
{'form': [('tests/fixtures/tribble/article.liquid', 34)]}

If a template's source text contains unbalanced block tags, those tags will appear in TagAnalysis.unclosed_tags.

from pprint import pprint
from liquid import Environment

env = Environment()

tag_analysis = env.analyze_tags_from_string("""\
{% for foo in bar %}
{% if foo %}
    {{ foo | upcase }}
{% endif %}
""")

pprint(tag_analysis.unclosed_tags)
# {'for': [('<string>', 1)]}

Tags that are not registered with the environment will appear in TagAnalysis.unknown_tags.

from pprint import pprint
from liquid import Environment

env = Environment()

tag_analysis = env.analyze_tags_from_string("""\
{% form article %}
  <h2>Leave a comment</h2>
  <input type="submit" value="Post comment" id="comment-submit" />
{% endform %}
""")

pprint(tag_analysis.unknown_tags)
# {'form': [('<string>', 1)]}

If an "inner" tag - like elsif and else for if and unless tags - appear without an appropriate enclosing block, those tags will appear in TagAnalysis.unexpected_tags.

from pprint import pprint
from liquid import Environment

env = Environment()

tag_analysis = env.analyze_tags_from_string("""\
{% for foo in bar %}
  {{ foo }}
{% endfor %}
{% break %}
""")

pprint(tag_analysis.unexpected_tags)
# {'break': [('<string>', 4)]}