def cycle(generator, args):
""" {% cycle v1 v2 v3 as varname %} """ # Not required to be nested inside a forloop, assigned to iterator
""" {% cycle varname %} """ # Iterator ++; and output
""" {% cycle v1 v2 v3 %} """ # cycle inside forloop.
if 'as' in args:
args, as_, varname = args[:-2], args[-2], args[-1]
generator.register_variable(varname)
generator.write(
'%s = _cycle(%s)' %
(varname, ','.join(map(generator.convert_variable, args))))
elif len(args) == 1:
varname, = args
if not generator.variable_in_current_scope(varname):
raise CompileException(
generator.current_tag,
'Variable %s has not been defined by a {% cycle %} declaration'
% varname)
generator.write('%s.next' % generator.convert_variable(varname))
else:
# How it works: {% for %} should detect wether some {% cycle %} nodes are nested inside
if not generator.variable_in_current_scope('forloop'):
raise CompileException(
generator.current_tag,
'{% cycle %} can only appear inside a {% for %} loop')
args = map(generator.convert_variable, args)
generator.write('sys.stdout.write([ %s ][ forloop.counter %% %i ])' %
(','.join(args), len(args)))
"""
def __init__(self, blocktrans):
# Build translatable string
plural = False # get true when we find a plural translation
string = []
variables = []
plural_string = []
plural_variables = []
for n in blocktrans.children:
if isinstance(n, DjangoPluralTag):
if not (len(blocktrans.params) and blocktrans.params[0].output_as_string() == 'count'):
raise CompileException(blocktrans,
'{% plural %} tags can only appear inside {% blocktrans COUNT ... %}')
plural = True
elif isinstance(n, DjangoVariable):
(plural_string if plural else string).append(convert_var(n.varname))
(plural_variables if plural else variables).append(n.varname)
elif isinstance(n, DjangoContent):
(plural_string if plural else string).append(n.output_as_string())
else:
raise CompileException(n, 'Unexpected token in {% blocktrans %}: ' + n.output_as_string())
# Return information
self.has_plural = plural
self.string = u''.join(string)
self.plural_string = ''.join(plural_string)
self.variables = set(variables)
self.plural_variables = set(plural_variables)
def handle_filter(subject, children):
filter_name = None
filter_option = None
in_filter_option = False
def result():
assert filter_name
if filter_name in _native_filters:
return _native_filters[filter_name](self, subject,
filter_option)
elif filter_option:
return '_f["%s"](%s, %s)' % (filter_name, subject,
filter_option)
else:
return '_f["%s"](%s)' % (filter_name, subject)
for i in range(0, len(children)):
part = children[i].output_as_string()
c = children[i]
if c.name == 'digits':
if not filter_option and in_filter_option:
filter_option = part
else:
raise CompileException(self._tag, 'Invalid variable')
elif c.name == 'name':
if not filter_option and in_filter_option:
filter_option = part
elif not filter_name:
filter_name = part
else:
raise CompileException(self._tag, 'Invalid variable')
elif c.name == 'string':
if not filter_option and in_filter_option:
filter_option = part
else:
raise CompileException(self._tag, 'Invalid variable')
elif c.name == 'trans':
if not filter_option and in_filter_option:
filter_option = '_(%s)' % c.output_as_string()
else:
raise CompileException(self._tag, 'Invalid variable')
if c.name == 'filter-option' and filter_name:
# Entered the colon ':'
in_filter_option = True
elif c.name == 'pipe':
# | is the start of a following filter
return handle_filter(result(), children[i + 1:])
return result()
def _compile_template(self, template, input_path):
try:
try:
# Open input file
code = codecs.open(input_path, 'r', 'utf-8').read()
except UnicodeDecodeError, e:
raise CompileException(0, 0, input_path, str(e))
except IOError, e:
raise CompileException(0, 0, input_path, str(e))
def _compile(n):
if isinstance(n, DjangoTag):
if n.tagname in __tags:
# Opening of {% tag %}
other_tags = __tags[n.tagname].tags
frame = TagFrame(n, n.tagname, other_tags, n.args)
frame.handler = __tags[n.tagname]
if other_tags:
stack.append(frame)
else:
top().append_content(frame)
elif stack and n.tagname == top().other_tags[-1].tagname:
# Close tag for this frame
frame = stack.pop()
top().append_content(frame)
elif stack and n.tagname in top().following_tagnames:
# Transition to following content block (e.g. from 'if' to 'else')
top().start_content_block(n.tagname)
else:
raise CompileException(
n, 'Unknown template tag %s' % n.tagname)
elif isinstance(n, DjangoVariable):
top().append_content(VariableFrame(n))
elif isinstance(n, basestring):
top().append_content(n)
elif isinstance(n, DjangoTransTag):
top().append_content(_(n.string))
elif isinstance(n, DjangoBlocktransTag):
# Create blocktrans frame
top().append_content(BlocktransFrame(n))
elif any([
isinstance(n, k)
for k in (DjangoPreprocessorConfigTag, DjangoComment,
DjangoMultilineComment, DjangoLoadTag,
DjangoCompressTag)
]):
pass
elif any([
isinstance(n, k)
for k in (DjangoContent, HtmlNode, DjangoRawOutput)
]):
# Recursively build output frames
n.output(_compile)
else:
raise CompileException(n, 'Unknown django tag %s' % n.name)
def read_media(url):
if is_remote_url(url):
try:
f = urllib2.urlopen(url)
if f.code == 200:
return f.read().decode('utf-8')
else:
raise CompileException(None,
'External media not found: %s' % url)
except urllib2.URLError, e:
raise CompileException(None,
'Opening %s failed: %s' % (url, e.message))
def process_template(self, template, input_path):
# TODO: when HTML processing fails, the 'context' attribute is not
# retreived and no translations are failed. so, translate again
# without html. (But we need to process html, in order to find
# gettext() in javascript.)
try:
try:
# Open input file
code = codecs.open(input_path, 'r', 'utf-8').read()
except UnicodeDecodeError, e:
raise CompileException(0, 0, input_path, str(e))
# Compile
output, context = compile(code, path=input_path, loader=load_template_source,
options=get_options_for_path(input_path))
for entry in context.gettext_entries:
line = '#: %s:%s:%s' % (entry.path, entry.line, entry.column)
if not entry.text in self.strings:
self.strings[entry.text] = set()
self.strings[entry.text].add(line)
if self.verbosity >= 2:
sys.stderr.write(line + '\n')
sys.stderr.write('msgid "%s"\n\n' % entry.text.replace('"', r'\"'))
def _preprocess_decorate_tags(tree, context):
"""
Replace {% decorate "template.html" %}...{% enddecorate %} by the include,
and fill in {{ content }}
"""
class DjangoPreprocessedDecorate(DjangoContent):
def init(self, children):
self.children = children
for decorate_block in list(tree.child_nodes_of_class(DjangoDecorateTag)):
# Content nodes
content = decorate_block.children
# Replace content
try:
include_tree = context.load(decorate_block.template_name)
for content_var in include_tree.child_nodes_of_class(DjangoVariable):
if content_var.varname == 'decorator.content':
content_var.__class__ = DjangoPreprocessedVariable
content_var.init(content)
# Move tree
decorate_block.__class__ = DjangoPreprocessedDecorate
decorate_block.init([ include_tree ])
except TemplateDoesNotExist, e:
raise CompileException(decorate_block, 'Template in {% decorate %} tag not found (%s)' % decorate_block.template_name)
def process_params(self, params):
param = params[1].output_as_string()
# Template name should not be variable
if param[0] in ('"', "'") and param[-1] in ('"', "'"):
self.template_name = param[1:-1]
else:
raise CompileException(self, 'Do not use variable template names in {% decorate %}')
def register_variable(self, var):
#if self.variable_in_current_scope(var):
if var in self._scopes[-1]:
raise CompileException(
self._tag,
'Variable "%s" already defined in current scope' % var)
else:
self._scopes[-1].add(var)
def _compile_template(self,
lang,
template,
input_path,
output_path,
no_html=False):
try:
# Create output directory
self._create_dir(os.path.split(output_path)[0])
try:
# Open input file
code = codecs.open(input_path, 'r', 'utf-8').read()
except UnicodeDecodeError, e:
raise CompileException(0, 0, input_path, str(e))
except IOError, e:
raise CompileException(0, 0, input_path, str(e))
def find_variables_in_function_parameter_list(nodelist):
# The `nodelist` parameter is the nodelist of the parent parsenode, starting with the 'function' keyword
assert isinstance(nodelist[0], JavascriptKeyword) and nodelist[0].keyword == 'function'
i = 1
while isinstance(nodelist[i], JavascriptWhiteSpace):
i += 1
# Skip function name (and optional whitespace after function name)
if isinstance(nodelist[i], JavascriptVariable):
i += 1
while isinstance(nodelist[i], JavascriptWhiteSpace):
i += 1
# Enter function parameter list
if isinstance(nodelist[i], JavascriptParentheses):
# Remember function parameters
variables = []
need_comma = False # comma is the param separator
for n in nodelist[i].children:
if isinstance(n, JavascriptWhiteSpace):
pass
elif isinstance(n, JavascriptVariable):
variables.append(n)
need_comma = True
elif isinstance(n, JavascriptOperator) and n.is_comma and need_comma:
need_comma = False
else:
raise CompileException(n, 'Unexpected token in function parameter list')
# Skip whitespace after parameter list
i += 1
while isinstance(nodelist[i], JavascriptWhiteSpace):
i += 1
# Following should be a '{', and bind found variables to scope
if isinstance(nodelist[i], JavascriptScope):
for v in variables:
add_var_to_scope(nodelist[i], v)
else:
raise CompileException(nodelist[i], 'Expected "{" after function definition')
else:
raise CompileException(nodelist[i], 'Expected "(" after function keyword')
def change(self, value, node=None):
"""
Change an option. Called when the template contains a {% ! ... %} option tag.
"""
actions = {
'compile-css': ('compile_css', True),
'compile-javascript': ('compile_javascript', True),
'disallow-orphan-blocks': ('disallow_orphan_blocks', True),
'html': ('is_html', True), # Enable HTML extensions
'html-remove-empty-class-attributes':
('remove_empty_class_attributes', True),
'merge-internal-css': ('merge_internal_css', True),
'merge-internal-javascript': ('merge_internal_javascript', True),
'no-disallow-orphan-blocks': ('disallow_orphan_blocks', False),
'no-html': ('is_html', False), # Disable all HTML specific options
'no-i18n-preprocessing': ('preprocess_translations', False),
'no-macro-preprocessing': ('preprocess_macros', False),
'no-pack-external-css': ('pack_external_css', False),
'no-pack-external-javascript': ('pack_external_javascript', False),
'no-validate-html': ('validate_html', False),
'no-whitespace-compression': ('whitespace_compression', False),
'pack-external-css': ('pack_external_css', True),
'pack-external-javascript': ('pack_external_javascript', True),
'validate-html': ('validate_html', True),
'whitespace-compression': ('whitespace_compression', True),
'no-block-level-elements-in-inline-level-elements':
('disallow_block_level_elements_in_inline_level_elements', True),
}
if value in actions:
setattr(self, actions[value][0], actions[value][1])
else:
if node:
raise CompileException(
node, 'No such template preprocessor option: %s' % value)
else:
raise CompileException(
'No such template preprocessor option: %s (in settings.py)'
% value)
def process_params(self, params):
param = params[1].output_as_string()
if param[0] == '"' and param[-1] == '"':
self.template_name = param[1:-1]
self.template_name_is_variable = False
elif param[0] == "'" and param[-1] == "'":
self.template_name = param[1:-1]
self.template_name_is_variable = False
else:
raise CompileException(self, 'Preprocessor does not support variable {% extends %} nodes')
self.template_name = param
self.template_name_is_variable = True
def _process_gettext(js_node, context, validate_only=False):
"""
Validate whether gettext(...) function in javascript get a string as
parameter. (Or concatenation of several strings)
"""
for scope in js_node.child_nodes_of_class((JavascriptScope, JavascriptSquareBrackets, JavascriptParentheses)):
nodes = scope.children
for i, c in enumerate(nodes):
# Is this a gettext method?
if isinstance(nodes[i], JavascriptVariable) and nodes[i].varname == 'gettext':
try:
gettext = nodes[i]
# Skip whitespace
i += 1
while isinstance(nodes[i], JavascriptWhiteSpace):
i += 1
# When gettext is followed by '()', this is a call to gettext, otherwise, gettext is used
# as a variable.
if isinstance(nodes[i], JavascriptParentheses) and not nodes[i].contains_django_tags:
parentheses = nodes[i]
# Read content of gettext call.
body = []
for node in parentheses.children:
if isinstance(node, JavascriptOperator) and node.operator == '+':
# Skip concatenation operator
pass
elif isinstance(node, JavascriptString):
body.append(node.value)
else:
raise CompileException(node, 'Unexpected token inside gettext(...)')
body = u''.join(body)
# Remember gettext entry
context.remember_gettext(gettext, body)
if not validate_only:
# Translate content
translation = translate_js(body)
# Replace gettext(...) call by its translation (in double quotes.)
gettext.__class__ = JavascriptDoubleQuotedString
gettext.children = [ translation.replace(u'"', ur'\"') ]
nodes.remove(parentheses)
except IndexError, i:
# i got out of the nodes array
pass
def parse_url_params(urltag):
if not urltag.url_params:
raise CompileException(urltag, 'Attribute missing for {% url %} tag.')
# Parse url parameters
name = urltag.url_params[0].output_as_string()
args = []
kwargs = { }
for k in urltag.url_params[1:]:
k = k.output_as_string()
if '=' in k:
k,v = k.split('=', 1)
kwargs[str(k)] = _variable_to_literal(v)
else:
args.append(_variable_to_literal(k))
return name, args, kwargs
def check_external_file_existance(node, url):
"""
Check whether we have a matching file in our media/static directory for this URL.
Raise exception if we don't.
"""
exception = CompileException(node,
'Missing external media file (%s)' % url)
if is_remote_url(url):
if urllib2.urlopen(url).code != 200:
raise exception
else:
complete_path = get_media_source_from_url(url)
if not complete_path or not os.path.exists(complete_path):
if MEDIA_URL and url.startswith(MEDIA_URL):
raise exception
elif STATIC_URL and url.startswith(STATIC_URL):
raise exception
def _preprocess_includes(tree, context):
"""
Look for all the {% include ... %} tags and replace it by their include.
"""
include_blocks = list(tree.child_nodes_of_class(DjangoIncludeTag))
for block in include_blocks:
if not block.template_name_is_variable:
try:
# Parse include
include_tree = context.load(block.template_name)
# Move tree from included file into {% include %}
block.__class__ = DjangoPreprocessedInclude
block.init([ include_tree ], block.with_params)
block.path = include_tree.path
block.line = include_tree.line
block.column = include_tree.column
except TemplateDoesNotExist, e:
raise CompileException(block, 'Template in {%% include %%} tag not found (%s)' % block.template_name)
def handle_var(children):
out = []
for i in range(0, len(children)):
part = children[i].output_as_string()
c = children[i]
if c.name == 'digits':
# First digits are literals, following digits are indexers
out.append('[%s]' % part if out else part)
elif c.name == 'dot':
#out.append('.') # assume last is not a dot
pass
elif c.name == 'string':
out.append(part)
elif c.name == 'name':
if out:
out.append('.%s' % part)
else:
if not self.variable_in_current_scope(part):
# If variable is not found in current or one of the parents'
# scopes, then prefix variable with "_c."
out.append('_c.%s' % part)
else:
out.append(part)
elif c.name == 'trans':
if out:
raise CompileException(self._tag, 'Invalid variable')
else:
out.append('_(%s)' % handle_var(c.children))
elif c.name == 'pipe':
# | is the start of a filter
return handle_filter(''.join(out), children[i + 1:])
return ''.join(out)
def found_missing():
raise CompileException(current_node(), 'Missing semicolon detected. Please check your Javascript code.')
def _process_extends(tree, context):
"""
{% extends ... %}
When this tree extends another template. Load the base template,
compile it, merge the trees, and return a new tree.
"""
extends_tag = None
try:
base_tree = None
for c in tree.all_children:
if isinstance(c, DjangoExtendsTag) and not c.template_name_is_variable:
extends_tag = c
base_tree = context.load(c.template_name)
break
if base_tree:
base_tree_blocks = list(base_tree.child_nodes_of_class(DjangoBlockTag))
tree_blocks = list(tree.child_nodes_of_class(DjangoBlockTag))
# Retreive list of block tags in the outer scope of the child template.
# These are the blocks which at least have to exist in the parent.
outer_tree_blocks = filter(lambda b: isinstance(b, DjangoBlockTag), tree.children)
# For every {% block %} in the base tree
for base_block in base_tree_blocks:
# Look for a block with the same name in the current tree
for block in tree_blocks[:]:
if block.block_name == base_block.block_name:
# Replace {{ block.super }} variable by the parent's
# block node's children.
block_dot_super = base_block.children
for v in block.child_nodes_of_class(DjangoVariable):
if v.varname == 'block.super':
# Found a {{ block.super }} declaration, deep copy
# parent nodes in here
v.__class__ = DjangoPreprocessedVariable
v.init(deepcopy(block_dot_super[:]))
# Replace all nodes in the base tree block, with this nodes
base_block.children = block.children
# Remove block from list
if block in outer_tree_blocks:
outer_tree_blocks.remove(block)
# We shouldn't have any blocks left (if so, they don't have a match in the parent)
if outer_tree_blocks:
warning = 'Found {%% block %s %%} which has not been found in the parent' % outer_tree_blocks[0].block_name
if context.options.disallow_orphan_blocks:
raise CompileException(outer_tree_blocks[0], warning)
else:
context.raise_warning(outer_tree_blocks[0], warning)
# Move every {% load %} and {% ! ... %} to the base tree
for l in tree.child_nodes_of_class((DjangoLoadTag, DjangoPreprocessorConfigTag)):
base_tree.children.insert(0, l)
return base_tree
else:
return tree
except TemplateDoesNotExist, e:
# It is required that the base template exists.
raise CompileException(extends_tag, 'Base template {%% extends "%s" %%} not found' %
(extends_tag.template_name if extends_tag else "..."))
def process_params(self, params):
self._params = params
if not len(self._params) == 3:
raise CompileException(self, '{% ifequal %} needs exactly two parameters')
def nest_block_level_elements(tree, mappings, _classes=Token, check=None):
"""
Replace consecutive nodes like (BeginBlock, Content, Endblock) by
a recursive structure: (Block with nested Content).
Or also supported: (BeginBlock, Content, ElseBlock Content EndBlock)
After execution, the first content will be found in node.children,
The second in node.children2
`_classes` should be a single Class or tuple of classes.
"""
check = check or (lambda c: c.name)
def get_moving_to_list():
"""
Normally, we are moving childnodes to the .children
list, but when we have several child_node_lists because
of the existance of 'else'-nodes, we may move to another
list. This method returns the list instace we are currently
moving to.
"""
node = moving_to_node[-1]
index = str(moving_to_index[-1] + 1) if moving_to_index[-1] else ''
if not hasattr(node, 'children%s' % index):
setattr(node, 'children%s' % index, [])
return getattr(node, 'children%s' % index)
for nodelist in tree.children_lists:
# Push/Pop stacks
moving_to_node = []
moving_to_index = []
tags_stack = [] # Stack of lists (top of the list contains a list of
# check_values for possible {% else... %} or {% end... %}-nodes.
for c in nodelist[:]:
# The 'tags' are only concidered tags if they are of one of these classes
is_given_class = isinstance(c, _classes)
# And if it's a tag, this check_value is the once which could
# match a value of the mapping.
check_value = check(c) if is_given_class else None
# Found the start of a block-level tag
if is_given_class and check_value in mappings:
m = mappings[check(c)]
(end, class_) = (m[:-1], m[-1])
# Patch class
c.__class__ = class_
# Are we moving nodes
if moving_to_node:
get_moving_to_list().append(c)
nodelist.remove(c)
# Start moving all following nodes as a child node of this one
moving_to_node.append(c)
moving_to_index.append(0)
tags_stack.append(end)
# This node will create a side-tree containing the 'parameters'.
c.process_params(c.children[:])
c.children = []
# End of this block-level tag
elif moving_to_node and is_given_class and check_value == tags_stack[-1][-1]:
nodelist.remove(c)
# Some node classes like to receive a notification of the matching
# end node.
if hasattr(moving_to_node[-1], 'register_end_node'):
moving_to_node[-1].register_end_node(c)
# Block-level tag created, apply recursively
# No, we shouldn't!!! Child nodes of this tag are already processed
#nest_block_level_elements(moving_to_node[-1])
# Continue
del moving_to_node[-1]
del moving_to_index[-1]
del tags_stack[-1]
# Any 'else'-node within
elif moving_to_node and is_given_class and check_value in tags_stack[-1][:-1]:
nodelist.remove(c)
# Move the tags list
position = tags_stack[-1].index(check_value)
tags_stack[-1] = tags_stack[-1][position+1:]
# Children attribute ++
moving_to_index[-1] += position+1
# Are we moving nodes
elif moving_to_node:
get_moving_to_list().append(c)
nodelist.remove(c)
# Apply recursively
nest_block_level_elements(c, mappings, _classes, check)
elif isinstance(c, Token):
# Apply recursively
nest_block_level_elements(c, mappings, _classes, check)
if moving_to_node:
raise CompileException(moving_to_node[-1].line, moving_to_node[-1].column, moving_to_node[-1].path, '%s tag not terminated' % moving_to_node[-1].__class__.__name__)
def _validate_javascript(js_node):
"""
Check for missing semicolons in javascript code.
Note that this is some very fuzzy code. It works, but won't find all the errors,
It should be replaced sometime by a real javascript parser.
"""
# Check whether no comma appears at the end of any scope.
# e.g. var x = { y: z, } // causes problems in IE6 and IE7
for scope in js_node.child_nodes_of_class(JavascriptScope):
if scope.children:
last_child = scope.children[-1]
if isinstance(last_child, JavascriptOperator) and last_child.is_comma:
raise CompileException(last_child,
'Please remove colon at the end of Javascript object (not supported by IE6 and IE7)')
# Check whether no semi-colons are missing. Javascript has optional
# semicolons and uses an insertion mechanism, but it's very bad to rely on
# this. If semicolons are missing, we consider the code invalid. Every
# statement should end with a semi colon, except: for, function, if,
# switch, try and while (See JSlint.com)
for scope in [js_node] + list(js_node.child_nodes_of_class(JavascriptScope)):
i = [0] # Variable by referece
def next():
i[0] += 1
def has_node():
return i[0] < len(scope.children)
def current_node():
return scope.children[i[0]]
def get_last_non_whitespace_token():
if i[0] > 0:
j = i[0] - 1
while j > 0 and isinstance(scope.children[j], JavascriptWhiteSpace):
j -= 1
if j:
return scope.children[j]
def found_missing():
raise CompileException(current_node(), 'Missing semicolon detected. Please check your Javascript code.')
semi_colon_required = False
while has_node():
c = current_node()
if isinstance(c, JavascriptKeyword) and c.keyword in ('for', 'if', 'switch', 'function', 'try', 'catch', 'while'):
if (semi_colon_required):
found_missing()
semi_colon_required = False
if c.keyword == 'function':
# One *exception*: When this is an function-assignment, a
# semi-colon IS required after this statement.
last_token = get_last_non_whitespace_token()
if isinstance(last_token, JavascriptOperator) and last_token.operator == '=':
semi_colon_required = True
# Skip keyword
next()
# and optional also function name
while isinstance(current_node(), JavascriptWhiteSpace):
next()
if isinstance(current_node(), JavascriptVariable):
next()
else:
# Skip keyword
next()
# Skip whitespace
while isinstance(current_node(), JavascriptWhiteSpace):
next()
# Skip over the '(...)' parameter list
# Some blocks, like try {} don't have parameters.
if isinstance(current_node(), JavascriptParentheses):
next()
# Skip whitespace
# In case of "do { ...} while(1)", this may be the end of the
# scope. Therefore we check has_node
while has_node() and isinstance(current_node(), JavascriptWhiteSpace):
next()
# Skip scope { ... }
if has_node() and isinstance(current_node(), JavascriptScope):
next()
i[0] -= 1
elif isinstance(c, JavascriptKeyword) and c.keyword == 'var':
# The previous token, before the 'var' keyword should be semi-colon
last_token = get_last_non_whitespace_token()
if last_token:
if isinstance(last_token, JavascriptOperator) and last_token.operator == ';':
# x = y; var ...
pass
elif isinstance(last_token, JavascriptOperator) and last_token.operator == ':':
# case 'x': var ...
pass
elif isinstance(last_token, JavascriptScope) or isinstance(last_token, DjangoTag):
# for (...) { ... } var ...
pass
elif isinstance(last_token, JavascriptParentheses):
# if (...) var ...
pass
else:
found_missing()
elif isinstance(c, JavascriptOperator):
# Colons, semicolons, ...
# No semicolon required before or after
semi_colon_required = False
elif isinstance(c, JavascriptParentheses) or isinstance(c, JavascriptSquareBrackets):
semi_colon_required = True
elif isinstance(c, JavascriptScope):
semi_colon_required = False
elif isinstance(c, JavascriptKeyword) and c.keyword == 'return':
# Semicolon required before return in: x=y; return y
if (semi_colon_required):
found_missing()
# No semicolon required after return in: return y
semi_colon_required = False
elif isinstance(c, JavascriptVariable):
if (semi_colon_required):
found_missing()
semi_colon_required = True
elif isinstance(c, JavascriptWhiteSpace):
# Skip whitespace
pass
next()
def _tokenize(node, nodelist, state_stack, token_stack, root=False):
"""
node: The current parse node that we are lexing. We are lexing
tokens in a parse tree of another language, and this node
is the parse node of the other language where we are now.
state_stack: The current state in our lexing 'grammar'
token_stack: The output token to where we are moving nodes right now.
This is a stack of childnodes lists
root: True when this is the main call.
"""
# Copy input nodes to new list, and clear nodelist
input_nodes = nodelist[:]
nodelist.__init__()
# Position TODO: this information is only right for the first children-list, not
# for the others!!!
line = node.line
column = node.column
path = node.path
# As long as we have input nodes
while input_nodes:
# Pop input node
current_input_node = input_nodes[0]
del input_nodes[0]
if isinstance(current_input_node, basestring):
# Tokenize content
string = current_input_node
# When the string starts with a BOM_UTF8 character, remove it.
string = string.lstrip(unicode(codecs.BOM_UTF8, 'utf8'))
# We want the regex to be able to match as much as possible,
# So, if several basestring nodes, are following each other,
# concatenate as one.
while input_nodes and isinstance(input_nodes[0], basestring):
# Pop another input node
string += input_nodes[0]
del input_nodes[0]
# Parse position
position = 0
while position < len(string):
for compiled_regex, action_list in states[ state_stack[-1] ].transitions():
match = compiled_regex.match(string[position:])
#print state_stack, string[position:position+10]
if match:
(start, count) = match.span()
# Read content
content = string[position : position + count]
# Execute actions for this match
for action in action_list:
if isinstance(action, Record):
if action.value:
token_stack[-1].append(action.value)
else:
token_stack[-1].append(content)
elif isinstance(action, Shift):
position += count
count = 0
# Update row/column
f = content.find('\n')
while f >= 0:
line += 1
column = 1
content = content[f+1:]
f = content.find('\n')
column += len(content)
elif isinstance(action, Push):
state_stack.append(action.state_name)
elif isinstance(action, Pop):
del state_stack[-1]
elif isinstance(action, StartToken):
token = Token(action.state_name, line, column, path)
token_stack[-1].append(token)
token_stack.append(token.children)
elif isinstance(action, StopToken):
# TODO: check following constraint!
# token_stack[-1] is a childnode list now instead of a node. it does no longer
# have an attribute name!
# if action.state_name and token_stack[-1].name != action.state_name:
# raise CompileException(line, column, path, 'Token mismatch')
del token_stack[-1]
elif isinstance(action, Error):
raise CompileException(line, column, path, action.message +
"; near: '%s'" % string[max(0,position-20):position+20])
break # Out of for
# Not a DjangoContent node? Copy in current position.
else:
# Recursively tokenize in this node (continue with states, token will be replaced by parsed content)
if isinstance(current_input_node, classes_to_replace_by_parsed_content):
for l in current_input_node.children_lists:
_tokenize(current_input_node, l, state_stack, token_stack)
# Recursively tokenize in this node (start parsing again in nested node)
elif isinstance(current_input_node, classes_to_enter):
for l in current_input_node.children_lists:
_tokenize(current_input_node, l, state_stack, [ l ], True)
token_stack[-1].append(current_input_node)
# Any other class, copy in current token
else:
token_stack[-1].append(current_input_node)
if root and token_stack != [ nodelist ]:
top = token_stack[-1]
raise CompileException(top.line, top.column, top.path, '%s not terminated' % top.name)
if f.code == 200:
return f.read().decode('utf-8')
else:
raise CompileException(None,
'External media not found: %s' % url)
except urllib2.URLError, e:
raise CompileException(None,
'Opening %s failed: %s' % (url, e.message))
else:
path = get_media_source_from_url(url)
if path:
return codecs.open(path, 'r', 'utf-8').read()
else:
raise CompileException(
None, 'External media file %s does not exist' % url)
def simplify_media_url(url):
"""
For a given media/static URL, replace the settings.MEDIA/STATIC_URL prefix
by simply /media or /static.
"""
if MEDIA_URL and url.startswith(MEDIA_URL):
return '/media/' + url[len(MEDIA_URL):]
elif STATIC_URL and url.startswith(STATIC_URL):
return '/static/' + url[len(STATIC_URL):]
else:
return url