This is what I would like in Sage 3.0

1. DOCUMENTATION:
   

   cd SAGE_ROOT/devel/sage/sage/; sage -coverage .

   
   should output
   

   Overall weighted coverage score:  x%  [[where x >= 50]]

   
   Moreover there should be at least one hand written paragraph at the beginning of each file; "sage -coverage" can be adapted to flag whether or not this is there. This will improve the overall quality and maintainability of Sage, and make it easier for users to find examples.
   
   
2. MANIPULATE: Usable "manipulate" functionality standard in Sage. This has very wide applicability.
   
   
3. R: A pexpect interface to R (so, e..g, the notebook can act as a full R notebook using 100% native R syntax). This will matter to a lot of Sage users, and make using R from Sage much easier in some cases (just cut and paste anything from any R examples and it will work). It will also provide something in Sage that one doesn't get with Python + rpy + R.
   
   
4. TIMING/BENCHMARKING: Fully integrate in Sage wjp's code that times all doctests, and start publishing the results on all Sage-supported platforms with every Sage release. This will give people a much better sense about which hardware is best for running Sage, and avoid major performance regressions. Likewise, get the Malb/wjp/my generic benchmarking code into Sage (this provides a doctest like plain text format for creating benchmarks, and is already mostly written).
   
   

I've only proposed goals for 3.0 that are wide reaching and will be noticed in some way by most users instead of fairly technical optimizations in a specific area (such as FLINT, Libsingular, or optimized integer allocation). I think changing implementations in specific technical areas to speed things up is more appropriate in week-to-week releases, and is also something we should be very careful about until we have good speed regression testing in place (we should have done step 4
above a long long ago).

Mathematics Research, Education, and Sage

I think there is a brewing tension between education and research amongst developers involved with the Sage project. More on that in a moment.

I cannot speak for all the core Sage developers, but I think I have some idea what some of them think and care about. My impression is that many of them are involved in Sage because they want to create software that they can use for attacking cutting edge research problems in their research area. This is true of me: I started Sage -- original called "Software for Arithmetic
Geometry Experimentation" -- to have a very powerful open software environment for computing with modular forms, abelian varieties,
elliptic curves, and L-functions.

I am quite happy that Sage has become much more general, addressing a huge range of mathematics, since this expands the range of good developers and also increasing the range of tools math researchers can bring to bare on attacking a problem results in better research. For example, the solutions
to many problems in number theory involve an incredible range of techniques in different areas of mathematics. I'm fairly certain that many of the people who have put in insane hours during the last few years making Sage what is it now (e.g., Mike Hansen, David Roed, Robert Bradshaw, David Harvey, Robert Miller, Emily Kirkman, Martin Albrecht, Michael Abshoff, etc.) have a similar perspective.

On the other hand, I teach high school students for a while every summer (in SIMUW), as do other people like David Roe, and of course I teach undergraduate classes... This is why I put so much effort into co-authoring things like the Sage notebook, which exist mainly to make the functionality of Sage more accessible to a wider range of people.

So, I think there is a brewing tension between education and research amongst developers involved with the Sage project (and in my case in my own mind). Some observations:

1. The research part of the Sage project is thriving and getting sufficient funding independent of any connection with educational applications of Sage. It very very healthy right now.

2. There is a lot of potential benefit to education in having a tool like Sage, since Mathematica is quite expensive, closed, etc. It's good for humanity
for Sage to be genuinely useful in an educational context.

3. People working on Sage for research have very limited time, and it can be frustrating being regularly asked to do things by the education community that not only have nothing to do with research, but are even sometimes at odds with it.

4. It is vitally important for the Sage project to be both well organized and have a clear sense of direction, purpose and goals.

It might be a good idea if the people who are really interested in Sage being a great tool for *education*, would consider doing the
following:

(a) setting up a mailing list called sage-edu for development discussions related to Sage in education. I realize that we just got rid of sage-newbie, but that was for a different reason -- because people were posting sage-support questions there and not getting responses.

(b) Gather together the best education-related tools in some sort
of organized package. This could start with Geogebra. I don't know. The key
thing is that there is no expectation at all that the people into Sage mainly for research do much of anything related to this project. I hope one outcome of this project would be an spkg that when installed would make available lots of cool extra stuff, and of course I would be very supportive about server space, posting of spkg's etc. And when this gets some momentum and
quality behind it this spkg would be included standard in Sage.

Basically I'm suggesting that everyone interested in making Sage the ultimate educational tool get organized, figure out who really wants to put
in an insane amount of effort on this sort of thing, and put together a bunch of cool tools. Stop thinking you have to convince a bunch of us research-focused people to do the work or that your ideas are good -- you
don't -- your ideas are good; it's just that if we put a lot of time into them we won't have time for our research.

Make an spkg that will be trivial to install into Sage and extend its functionality. There is definitely sufficient interest in something like this
for education, there is great potential for funding, and potential for having a major positive impact on society. Thus I think people will emerge who will
want to take up this challenge. I just thing it's better if it can happen for a while unconstrained by the rules or prejudices of the "Sage Research" side
of this project.

In summary, please put a huge amount of effort into getting organized and putting together something polished and great, so I can later effortless assimilate it :-).

Benchmarketing: Modular Hermite Normal Form

Two days ago there was no fast freely available open source implementation of computation of the Hermite Normal Form of a (square nonsingular) matrix. In fact the fastest implementations I know of -- in Gap, Pari, and NTL, are all massively slower (millions of times) than Magma. See Allan Steel's page on this algorithm.

I just spent the last few days with help from Clement Pernet and Burcin Erocal implementing a fast modular/p-adic Hermite Normal Form algorithm. As soon as the coefficients of the input matrix get at all large our implementation is now the fastest in the world available anywhere (e.g., faster than Magma, etc.). This is incredibly important for my research work on modular forms.

The following plot compares Sage (blue) to Magma (red) for computing the Hermite Normal Form of an nxn matrix with coefficients that have b bits. The x-axis is n, the y-axis is b, and the blue dots show where Sage is much better, whereas the red dots are where Magma is much better. The timings are on a 2.6Ghz core 2 duo laptop running OS X, and both Sage and Magma were built as 32-bit native executables:



(Note: When doing timings under Linux, Magma doing slightly better than it does above, though Sage is still much better as soon as the coefficients are at all large.)

This plot shows the timings of Magma (red) versus Sage (blue) for computing the Hermite Forms of matrices with entries between -2¹² and 2¹². Sage is much faster than Magma, since the blue line is lower (=faster).


For even bigger coefficients Sage has a much bigger advantage. E.g., for a 200 x 200 256-bit matrix, Magma takes 232 seconds whereas Sage takes only 55 seconds.

In order to make this happen we read some old papers, read notes from a talk that Allan Steel (who implemented HNF in Magma) gave, and had to think quite hard to come up with three tricks (2 of which are probably similar to tricks alluded to in Steel's talk notes, but with no explanation about how they work), coded for 3 days at Sage Days 7, discovered that we weren't right about some of our tricks, doing lots of fine tuning, and finally fixed all the problems and got everything to work. We will be writing this up, and of course all the code is open source.

Acknowledgement: Many many thanks to Allan Steel for implementing an HNF in Magma that blows away everything else, which gave us a much better sense of what is possible in practice. Thanks also to the authors of IML whose beautiful GPL'd implementation of balanced p-adic is critical to our fast HNF implementation.

Josh Kantor's Lorenz attractor example

Josh posted a nice example of plotting a Lorenz attractor in Sage:


Put this in a notebook cell (be careful about newlines):

Integer = int
RealNumber = float

def lorenz(t,y,params):
    return [params[0]*(y[1]-y[0]),y[0]*(params[1]-y[2])- y[1],y[0]*y[1]-params[2]*y[2]]

def lorenz_jac(t,y,params):
    return [ [-params[0],params[0],0],[(params[1]-y[2]),-1,-y[0]],[y[1],y[0],-params[2]],[0,0,0]]

T=ode_solver()
T.algorithm="bsimp"
T.function=lorenz
T.jacobian=lorenz_jac
T.ode_solve(y_0=[.5,.5,.5],t_span=[0,155],params=[10,40.5,3],num_points=10000)
l=[T.solution[i][1] for i in range(len(T.solution))]

line3d(l,thickness=0.3, viewer='tachyon', figsize=8)

and this is what you get (click to zoom):

I'm glad I chose Python for Sage -- some cool scientific computing Python projects

When brainstorming for talk ideas for a workshop, Fernando Perez came up with a bunch of random off-the-top of his head very high quality/impact scientific computing projects that involve Python. Here they are (the following was written by Fernando Perez, the author of IPython):

- I can obviously talk about ipython and related projects, but I can also give a math/technical talk off this type of work (the whole implementation is python, and uses quite a few tricks):

http://dx.doi.org/10.1016/j.acha.2007.08.001

- Brian Granger (from Tech-X http://txcorp.org) has a NASA grant to develop distributed arrays for Python, using IPython and numpy. That would make for an excellent talk, I think (matlab, interactivesupercomputing.com, and all the 'big boys' are after distributed arrays).

- Trilinos (http://trilinos.sandia.gov), a large set of parallel solvers developed at Sandia National Lab, has a great set of Python bindings (even usable interactively via ipython).

- MPI4Py is an excellent set of MPI bindings for Python, and its author Lisandro Dalcin is also the developer of Petsc4py:
* http://mpi4py.scipy.org/
* http://code.google.com/p/petsc4py/
If Lisandro can't come, I can contact one of the Petsc guys who's a great python developer and see if he's coming or can make it, he's an excellent speaker (Matt Knepley from Argonne Nat. Lab http://www-unix.mcs.anl.gov/~knepley/).

- The Hubble space telescope people are all pyhton based, and have done enormous amounts of work on both their internal image processing pipeline and contrbuting to Matplotlib (they have currently a developer working full time on matplotlib).

- The NetworkX (Sage uses this) guys from Los Alamos will probably be coming: https://networkx.lanl.gov/wiki.

- The Scripps institute has an extremely strong Python team doing molecular visualization and visual programming. Their work is very impressive, and they're already in San Diego, so it's a no brainer to attend. Their presentations are always of very high quality: http://mgltools.scripps.edu.

- JPL uses python extensively (they've contracted out work for matplotlib to be extended to suit their specific needs).

- My new job at UC Berkeley is on neuroscience, and we could present the work that's being developed there for fMRI analysis (all Python based, fully open source, NIH funded).

- Andrew Straw's work (http://www.its.caltech.edu/~astraw/) on real-time 3d tracking of fruit flies is very, very impressive. All python based, hardware control, real-time parallel computing.

- The CACR group at Caltech has the contract for DANSE (http://wiki.cacr.caltech.edu/danse/index.php/Main_Page), the Spallation Neutron Source's data analysis framework, all python. This is currently the largest experiment being funded in the USA.

AMS Meeting Day 2: The Competition

Today I violated the Mathematica license agreement in front of Eric Weisstein (a famous Mathematica developer), I talked with the developers of Wiris (rhymes with virus) which is a commercial competitor to the Sage notebook, discussed Mupad with a longtime Mupad developer, and gave away $1000 of tutorials, DVD's, and other goodies at the Sage AMS exhibit booth.

There is a project called "Wiris", which I had never heard of until today. So Tom Boothby and I had a very interesting talk with the people at the Wiris Booth (http://www.wiris.com/). Wiris is a closed-source commercial math software company in Barcelona that makes a web-based interface to their own custom mathematical software (interestingly, one of their main developers took Calculus from Jordi Quer, who wrote a lot of modular-forms related code for Sage recently). Their primary audience and market right now is European Government Agencies who use their software for high school and beginning college education. Their software is much different than the Sage notebook, since it is written entirely in Java instead of being an AJAX javascript application. They knew about Sage and asked if it used OpenMath or MathML, and I explained that it didn't use either, that it shouldn't and that for our purposes (i.e., interfacing math software) those technologies do not solve the problems we have -- in fact, they are worse than useless. They said that us not taking the OpenMath route was disappointing. They ended the discussion by telling us that their web-based interface is much better than ours :-). I guess they were trying to intimidate us.

Eric Weisstein -- who told us that he is now an official Mathematica developer (doing graph theory among other things) in addition to his "Math World" came over to the Sage booth and asked a lot of pointed questions about Sage, mostly "how does this get funded?" We ended up talking for about an hour. There are numerous people at Mathematica who are well aware of Sage, and he claims he doesn't see Sage as a threat to Mathematica as a company. He said Mathematica has had a recent explosion in hiring as a result of greatly increased sales because of the new "Demonstrations" feature in Mathematica 6. We also talked a lot about graph theory and how Sage has a complete implementation of graph isomorphism testing, etc., which greatly impressed him (thanks Robert Miller!). Jason Grout asked about the Mathematica end use license agreement and Eric pleaded IANAL and I demonstrated using Mathematica via the Sage notebook -- all locally over localhost on a machine with a valid Mathematica license -- that this violates the Mathematica license. He got annoyed when I did a Mathematica graph through the Sage notebook... Later Eric talked about how he "hoped" Sage would continue to have momentum and not just die like other free projects. He gave Maxima as an example of a dead project, and seemed quite shocked when I mentioned that they are very much alive and have regular releases, etc. He then said that there is no such thing as free.

I met somebody from MuPad who has worked on that closed source project for a decade. They used to be a German national government funded project, then had an academic research branch until one year ago, and now are 100% commercial and private. I asked about their vision for the next 5 years, and he said "MuPad will survive", and said they were mainly happy to be stable. They also mentioned wanting to do more numerical and applied functionality.

I talked with a Unix guy who works on some Scientific Workplace. There is only a Windows version, and he is working on doing an OS X/Linux port, which they will finish "in the future"!

And that's just a little of what happened today...

AMS Meeting Part 1

I am completely exhausted right now, having put in a huge amount of time with little sleep during the last few days writing 3d graphics code for Sage (with Robert Bradshaw), making the Sage-2.9.2 release (with Michael Abshoff), printing fliers, tutorials, and preparing DVDs (with my brother) for the massive joint AMS meeting where there will be a Sage exhibit booth.

Our booth with have a huge banner that says "Creating a viable free open source alternative to Magma, Maple, Mathematica, and Matlab."
An Austrian named Harald Schilly did most of the work creating this poster.

I have no clue what to expect during the next few days. I've never run an exhibit booth before; I'm a mathematician not a "vendor", and my product is free. This should be very interesting. I've been to numerous AMS meetings before, and every booth I remember is commercial... TI, book publishers, math games, Maple, Mathematica, etc.