Millipede - Give your snake legs!
======

Millipede is a Python application compiler.


Goal
----

Python is a blazingly fast language, but only if the programming task is abstract enough to make use of the full suite of Python's features -- e.g. a program written in C at a similar level of abstraction would have similar speed.  Python appears slow compared to many other languages in benchmarks largely because many/most programming tasks (particularly in benchmarks) do not take advantage of the majority of Python's features.  The goal of Millipede is to have our cake and eat it too -- to provide the full power of Python when it's called for and to automatically remove unused abstractions for maximal performance everywhere else.


How we do this:
---------------

1) Static analysis --
We use offline analysis of a project's source to pull out semantic information about how it uses Python.  Unfortunately, in order to get any level of useful knowledge out of a project's source we need to do whole-program analysis; furthermore, doing this sort of static analysis is slow.  The upshot is that this works best if we compile a full project at once, rather than doing the analysis one-file-at-a-time.  This is the one major incompatibility between Millipede and Python.  However, it's not a total lost cause, since importing files we haven't analyzed simply emits a PyImport_ImportModule(...) and runs them in the normal Python interpreter loop.  Therefore, we can still be useful for programs that have a central core of functionality and add (at runtime) extensions or plugins to that base.

2) Feedback from the Analysis --
If compilation is a completely opaque process, it is hard for a programmer to understand why a certain bit of code is compiled the way it is.  Millipede will integrate a GUI that shows the results of static analysis and tag that analysis with the information a programmer needs to optimize their code.  For example, if a module optionally aliases a builtin in a global variable, then using that builtin in a tight inner loop will be slow.  This is something that is easy for a programmer to fix (e.g. by assigning the builtin to a local), but only if they notice the problem in the first place.  

It should be both fast and easy to scan a piece of code for potential trouble spots.  We see this becoming something like predator vision modes for code: the highlight color of a name will be based on the specificity of their derived type, the amount of indirection required for accesses, etc.

3) Feed-forward to the Compiler --
Some semantic aspects of a program are deviously hard to work out (in a reasonable amount of time) in a static analyzer.  For these cases (particularly in performance critical code) it may be helpful for a programmer to give hints to the compiler, based on what they see in the GUI.  If these appear as documentation (e.g. type annotations) in a form that is also useful to the human reader, then the cost to build an maintain them will hopefully be small enough to be worth the performance win.  

Note well:
Turning Python into C++ is not the goal!  We want our full, type-agnostic dynamism, but we need our performance too.  In cases where we can get that performance automatically, wonderful.  That said, pragmatism beats purity and if we can get a solid speed improvement by adding the occasional type annotation, so be it.


What about JITing?
------------------
A tracing JIT is simply a code execution cache.  This is a planned feature for Millipede.


Current Status
--------------
Millipede is fully self hosting, and it's performance in most cases is roughly equal in speed to CPython.

Halfway There: The GUI elements exist (built on PyQt4) although the highlighting engine is still mostly pointing at a prior experiment and are, thus, mainly broken at the moment.

Halfway There: Many optimizations based on control flow and type analysis.


Playing with Millipede
-------------------
For the truely brave/foolhardy:

> python3 run.py -P 3.1 -O asp path/to/translate.py
### This will spit out some stuff and maybe create a test.c, if you are lucky.
### Good candidates of things that are likely to work are in test/<foo>/<bar>.py
> make
### This makefile will try to link against python3.3 (e.g. cpython tip).
### It will also work with 3.1 and 3.2, so just twiddle the Makefile if needed.
> ./test-prog
### And check the output against the code, if you value correct answers.

Or run all tests with:
> py.test -k <interpreter>:<version>:test/ -v test/
### <Interpreter> is one of "python", "millipede", or "hosted" and <version> is one of 3.1 or 3.3.