Hi, and welcome to GitHub!

This looks good. I see you also made some formatting and grammar corrections, which is nice.

Optionally, if you want, there are some design notes also in the macro readme, which could be moved into design-notes.md. See the section titled Meta. Of the information currently in that section, the macro readme needs just a short description of the xmas tree combo, the rest are design notes. (Though maybe the "this is semantics, not syntax" disclaimer could be left in. Or maybe it could be moved. I'll leave it for you to decide which you think is better.)

Also, when reviewing the changes, I noticed that maybe it would be better if the silly attempts at Python humor (eels etc.) in the original text were moved out of the headings and into the main text, but technically that's an issue separate from the actual splitting. Basically, to improve usability, the heading being jumped to should match the text visible on the link.

If you feel this is already enough, I can merge the PR as-is - just post a comment here to tell me which option you prefer.

Hello! And thanks.
I think you've made some good suggestions, so I'll work on integrating those.

Thanks for the updates, looking good!

I wasn't aware GitHub-flavored Markdown supported expandable sections - that's exactly what this documentation needed. Nice!

I see you've also removed the added/changed notes, which is good - we can tick off another item from the TODO list in #2.

Actually, I'm thinking that since the pace of development has slowed somewhat, and with the polishing of the documentation, the next release (currently codenamed 0.14.2) could be called 1.0. Then it would be possible to start using semantic versioning.

Tell me when you're ready with the changes, I'll review and merge then.

Thank you! And apologies for the commit storm, I'm still getting used to contributing.

So far I've looked through the macros document and the design notes, but I'd like to go through the pure-python document (and everything else one more time) before finally merging.

Thank you for your patience!

A commit storm is a regular day in a git repo... :)

Ok, I'll wait until you say it's ready.

If you want to combine (some of?) the commits into a single commit, this can be done before merging. Look into interactive rebasing, especially how to squash commits. But be careful, it is possible to make mistakes during the process that may permanently erase commits with no way to restore them.

I've done it myself using Magit (the git frontend for Emacs; squashing instructions), but it should also be possible from the git command line. Git-scm.com has a section on interactive rebasing. There's also a general explanation of rebasing in their Pro Git book (it's free to read).

If there are just a few commits, I usually don't bother squashing them, unless the project maintainer specifically requests it. Different projects on GitHub have different guidelines. Large, popular projects obviously have to be picky to keep their commit history manageable, but for small projects, the commit volume is so low anyway that often it doesn't matter that much.

Of course, then there's the viewpoint of semantics - many developers prefer that a commit identifies one meaningful set of changes. This makes it easier to select just those changes for rebasing later, should a need arise e.g. to backport a bugfix or feature onto an earlier version of the codebase. (Then you just pick that one commit and you're done. Otherwise, will have to dig through the commit log to find which ones to take.) But unpythonic only has a master branch, so it doesn't really matter.

I'll leave the choice up to you. :)

(Magit is great, but perhaps Emacs has a bit too much of a learning curve if you don't need it otherwise.)

I closed the separate TODO issue I opened yesterday for merging the documentation improvements - let's just use this PR to track the status.

So don't worry, even though the referring issue above is closed, the improvements are going to be merged into 0.14.2 (or 1.0 as it'll probably be called) :)

Looking at the milestone, there's just one more code enhancement to make, after which the final tasks for the upcoming release are to merge your documentation improvements (once complete), and to document the new features and changes. (No hurry - the release schedule is "when it's ready".)

Some new functions have been added, and islice has been extended to accept negative start and/or stop. When it's time, I'll scrape the commit log for a full list.

I can write the initial version of the new parts of the documentation. If you want, proofreading is welcome - this would ensure it gets the same QA treatment as the rest - or if you feel that's too much, we can just leave the new parts at my version for now.

I've looked through everything and am fine with the changes for now.

I can proofread the new parts of the documentation! I'll wait on standby until you're ready.

Great!

The next step is for me to review and merge this PR.

It's more convenient to handle the proofreading in a separate PR, so I can easily make my updates on top of the already reformatted documentation. I'll post a comment here (after merging and closing) to let you know when I have the new parts ready.

I looked at the changes and mostly the PR looks great!

I have some minor issues for which I'll be requesting changes - tell me if you want to make the changes yourself, or leave them for me to make when I merge. (I'll make a detailed list as soon as I can, and send it as a review here.)

One important point I noticed when looking at the changes is that I see now my original documentation may not have been clear enough on the overall layout of unpythonic, so let me explain. :)

The unpythonic library consists of two parts - the pure-Python layer and the macro layer. It is correct that the pure-Python layer acts as a core for the macro layer, but that's not the full story.

What I had missed to point out clearly is that the pure-Python layer also provides some functionality, which has no macro equivalent. Those parts of the pure-Python layer are intended to be used directly.

As someone once put it, macros are the nuclear option of software engineering. In order to keep your design clean and elegant, the general guideline is to avoid writing a macro whenever a regular function could perform the same job. However, there are some jobs regular functions cannot perform. Some language extensions can be extremely inconvenient to use or even impossible to build without macros. This is the role of the macro layer.

For example, continuations is one such feature. There is a library by Thomas Baruchel for manual continuation-passing, but I think automating the CPS conversion makes the code much more readable.

(Though to be fair, I don't know whether Python needs continuations - Paul Graham happened to provide a recipe in his book, and it was fun to build something similar in Python, so I just went ahead and did it. The general opinion is that continuations are a feature of which much more is written about than it is actually used. Python already has a limited suspend-and-resume facility in the guise of generators and coroutines. It's difficult to come up with an actual use case where the resume multiple times from an already executed point of your code feature of general continuations makes a crucial difference.)

As for the let and do constructs, it is technically possible to use them without macros, but the macro version is so much more convenient to use, that I'm tempted to call the pure-Python version an implementation detail. (In particular, the macro version hides the unwieldy environment parameter, as well as performs lexical scoping automatically.)

Some features are borderline cases. For example, TCO can be used without macros, but you have to remember to decorate each function with @trampolined manually, and to return jump(f, ...) instead of return f(...) to make a tail call. The tco macro writes these parts for you (incidentally, this is what Lisp programmers mean when they talk about programs that write programs). In such a case, whether or not to rely on macros, I think is a trade-off each user must decide for themselves.

At the other extreme, the "batteries" for itertools and functools are designed to be used as-is - no macros needed. If these features are all the user wants from unpythonic, then MacroPy is not needed.

Having only a soft dependency on the macro system is for both cultural and technical reasons. To most programmers outside the Lisp community, macro systems are unfamiliar, and not understood very well. The title of Greg Hendershott's macro tutorial for Racket I think summarizes the situation well.

In the case of Python, there is also the technical aspect that Python was not designed to have macros. Specifically, there are no backward compatibility guarantees regarding its AST representation. In fact, some parts of the AST have changed several times during the 3.x series (see GTS). So in Python, depending on a package that relies on the AST structure may cause your software to break later, when you upgrade to a later minor release of Python (3.x → 3.y), if the macro package happens to go out of support. This includes MacroPy itself, as well as any packages that depend on it. For projects that must remain runnable in the long term with minimal maintenance, this is a concern.

So, I think the main points of this should be explained in the documentation, particularly the fact that the pure-Python layer essentially consists of two parts - underlying implementations for macros, and features that are meant to be used directly.

Also, it should be marked clearly which is which. I suppose this is marking is something for me to do, since I designed the thing. :)

Ah, one more thing that I didn't yet explain clearly - there are three kinds of features in unpythonic, not just two:

1. Features that only exist in the pure-Python core, and are meant to be used directly (e.g. batteries for itertools),
2. Features that have a pure-Python core, plus a macro layer that improves the user experience (e.g. do and let), and
3. Features that only exist as macros (e.g. continuations, lazify and dbg).

Item 3 was missing from the previous explanation.

This classification still leaves some sporks - for example, curry itself can be used manually, but automatic currying is only available as the with curry macro - but perhaps it's close enough as a short general explanation.

Thanks for reviewing! I'll look at the changes you've raised, and look into adding the distinction between the three sets of features.

Great!

Looking forward to the update!

Alright! I've finished up the changes and adding further explanation about the types of features offered.

Great work. Merged!

Thanks for your contribution!

Thank you! If you have any more documentation needs, feel free to ping me :-)

Ok, I'll drop you a note here when I have finished the documentation for the new features for 0.14.2. :)