Wednesday, December 10, 2008
I have been asked by EnterpriseDB to help port Red Hat's Spacewalk to Postgres; currently it runs only on Oracle. EnterpriseDB and Red Hat employees will be involved, but I expect Spacewalk and perhaps Postgres community members to be involved too because all the work will be done in an open source fashion. (Spacewalk is the community version of Red Hat Network Satellite).View or Post Comments
Saturday, December 6, 2008
Having traveled a week a month for the past twelve months, and having analyzed my email reading pattern, I have decide to cut back on my travel to focus on my community housecleaning duties and to help finalize Postgres 8.4. I will still attend some events but will reduce my travel so my community work doesn't suffer. There are already excellent speakers attending events so my absence should have a minimal impact.View or Post Comments
Tuesday, December 2, 2008
After describing how often I read my Postgres email, I would like to explain how I process email, which is basically done in two steps. The first step is to read every Postgres email message; many of them can be read, perhaps replied to, and discarded, e.g.:
|o||questions answered by someone|
|o||invalid bug reports|
|o||valid bug reports fixed by someone|
|o||requests for advice|
This allows me to delete ninety percent of email from my mailbox immediately. The remaining email must be kept, e.g.:
|o||valid or undetermined bug reports|
|o||patch submissions or discussions|
|o||feature requests or suggested improvements|
For these emails, I have to wait for an event:
|o||the question answered|
|o||the bug report confirmed, fixed, or rejected|
|o||the patch recorded, applied, or rejected|
|o||the feature or improvement request accepted or rejected by the community|
Right now I have 1,800 emails that are in this later state. What I do when I am not reading new email is to cycle through the saved email to see if they can be deleted or if I can resolve the issue myself. I also respond to email threads to help where I can.
The commitfest wiki has greatly relieved me of having to manage the patch review/application process, but I still track open items to make sure that, before each major release, every bug has been either fixed or added to the TODO list, and every submitted patch has been either applied or rejected.View or Post Comments
Monday, December 1, 2008
As a frequent traveler, I have always struggled with reading Postgres email during trips. There are basically two ways to do it: First, I can read email during the day, usually while attending a conference; the downside is that while I am reading email I am not interacting with anyone else, so I might as well have stayed home and not even attended. The second way is to read email in my hotel room at night; this works if most of the conference attendees are local and therefore go home at night, but if they are in hotels I go out with them, or again, I might as well not have attended. This is usually why I have one thousand Postgres emails to read when I return from a trip.
Reading my personal email during trips is never a problem because I only get a handful of personal emails a day.View or Post Comments
Thursday, November 20, 2008
When you have been tracking Postgres email as long as I have, you start to ask strange questions, like how often does new email arrive? How quickly do I read email? Well, I finally decided to get some numbers to answer those questions. The top of my Postgres blog page now has statistics on how many unread email message I currently have, as well an historical graph of unread emails and an hourly graph of email traffic.
For example, I currently have three unread Postgres email messages, but that count was higher yesterday because I had been busy the previous seven days with a visit to EnterpriseDB, a vacation day, a three-day conference, and two days updating my replication talk. I expect the unread email graph to spike in the next two weeks because I am busy tomorrow, have a marriage conference this weekend, am on a family vacation in Boston the next week, and then I am in India for a week to speak for EnterpriseDB at a conference and visit the EnterpriseDB Pune office.
Update 2: I have cancelled my trip to India because it would have caused my family too much worry.View or Post Comments
Monday, November 17, 2008
I returned from OpenSQL Camp yesterday. There were about sixty people in attendance, mostly MySQL folks, but with representatives from all open source databases, including Postgres, SQLite, and Firebird. Greg Sabino Mullane and I were the Postgres community attendees. There was lots of sharing about the challenges common to all database systems.
Two talks really interested me. The first was by SQLite author Richard Hipp where he explained how SQLite accomplishes atomic commits without a server process, using only file system operations.
The second interesting talk was about the columnar MySQL database storage engine Infobright. There has been lots of buzz about columnar databases, but this talk described how it all works, and the advantages. The second page of this PDF has an illustration: when data is loaded, rows are split into columns, and column values are stored together, rather than with the other values in the same row.
For example, the diagram shows four columns of cubes, representing a table with four columns, where each cube holds 64k column values. Row ordering is preserved so the first value of each top cube represents the first row in the table. Each cube is compressed, and because the compression happens among values in the same column, compression is very effective.
The second interesting thing about Infobright is how queries are executed. Each cube has statistical information, similar to Postgres optimizer statistics, but the statistics are completely accurate, not estimates. When a query is executed, these statistics are used to determine if there is any value in the cube which could potentially match the query. Effectively, this means every column has an index with 64k value granularity.
Of course, while Infobright can probably handle INSERT, UPDATE, and DELETE, these are always going to be expensive operations. Infobright is designed for read-only queries and bulk inserts, e.g. data warehousing.View or Post Comments
Monday, November 3, 2008
I have decided to attend OpenSQL Camp in Charlottesville, Virginia, an event designed to bring together developers from multiple open source databases. If others want to attend, I suggest you sign up soon because the soft deadline is tomorrow, November 4.View or Post Comments
Tuesday, October 14, 2008
I just returned from PG West in Portland, Oregon. It was a good event, with about eighty people attending. It took place at an ideal venue, a Portland State University building that was close to good restaurants and a park; we used the entire building.
The talks were of high quality, highlighting the increased seriousness of our developers. Two talks particularly interested me. The first was "The PostgreSQL Optimizer Exposed", by Tom Raney. It covered many topics related to the optimizer — topics I have not seen covered in other talks. The second interesting talk was "PL/LOLCODE HOWTO", by Josh Tolley. While I am not a LOLCODE fan, the talk covered many of the intricacies of adding a procedural language to Postgres. David Fetter's talks about recursive and windowing functions were good too. I also debuted my "Inside PostgreSQL Shared Memory" talk at this event.View or Post Comments
Saturday, October 4, 2008
During my recent downtime, I learned Perl. I have always used either C or shell script for most of my development; for me, Perl seemed in the middle of those two, feature-wise. I have considered learning Perl in the past, but every time I read a Perl book I failed to see Perl's usefulness for me. However, I came upon "Learning Perl the Hard Way" (PDF) and this short, high-level ebook got me excited about Perl.
I have converted two of my most complex shell scripts to Perl; as shell scripts, they were slow and hard to maintain. The rewritten Perl scripts are 200-400 lines long (about the same length the original shell scripts) and 15-25 times faster because of the improved algorithms possible in Perl and reduced subprocess creation.
I hope to use Perl from now on for Postgres developer scripts. We already have a few Perl scripts in pgsql/src/tools and I hope to expand on that, not only for maintainability and improved features, but also to allow some Windows users to run the scripts.View or Post Comments
Saturday, October 4, 2008
I am resuming my travel schedule, so I will be attending PostgreSQL Conference West in Oregon next week. I plan to focus on outstanding patches during October in preparation for the final 8.4 commit-fest in November.View or Post Comments
Tuesday, September 16, 2008
My father-in-law died on Sunday. I should be back to full-time work next week, and I will consider resuming my travel schedule in early October. Though my father-in-law never attended an open source conference, he did meet the many Postgres community members who visited my home.View or Post Comments
Thursday, September 11, 2008
I have just informed event organizers that I cannot attend the Brazilian PostgreSQL Conference in late September. My father-in-law, who has lived with me since I got married in 1989, is terminally ill and I must help my family care for him. EnterpriseDB has been very supportive of my need for personal time.
My events page looks like a train departure board in a blizzard because of all the cancellations; I will get back on track once things stabilize at home. I am still reading my Postgres email, just not as promptly.View or Post Comments
Thursday, September 4, 2008
Postgres chose an elephant mascot in 1997 because of the saying, "Elephants never forget". Research now shows that elephants can count too.View or Post Comments
Saturday, August 30, 2008
I just gave my final presentation at Brazil's first open source e-government conference, CONSEGI 2008 (news report). I have never seen so much enthusiasm for open source and Postgres as I have seen here in Brasilia. Of course, the open source people don't feel open source is being adopted fast enough, but I have felt that way about Postgres since 1996, so this is to be expected. ;-)
The Brazilian government has embraced open source more than any other government I have seen. How many open source conferences have cocktail parties at the Minsistry of Foreign Affairs? Several banks and Brazilian ministries also signed an agreement to use the Open Document Format (ODF) as supported by Open Office rather than continue to use Microsoft's closed proprietary document formats.
The event organizers also scheduled various cultural events during the conference, including a play, a band, a puppet show, and performances by an indigenous tribe; it really enhanced the conference.
I have meetings at several government ministries early next week to discuss Postgres usage.
UPDATE: I ended up giving eleven presentations in Brasilia.View or Post Comments
Thursday, August 28, 2008
There has been much discussion within the Postgres community about having a kitchen sink distribution of Postgres that bundles the database server with the many plug-ins available for Postgres. We have had something similar for Windows since 2005 but not for other operating systems. Such distributions are helpful for new users and people wanting to experiment with Postgres and its external enhancements.
It looks like Postgres Plus might be that distribution. I casually mentioned Postgres Plus to someone I met in Brasilia on Tuesday. On Wednesday they used it for a new installation of Pentaho, and on Thursday they said how impressed they were with Postgres Plus.
I have gotten several similar reports in the past month, even from Postgres veterans, which might mean that Postgres Plus is on its way to becoming our solution for super-easy installs.View or Post Comments
Wednesday, August 20, 2008View or Post Comments
Wednesday, August 20, 2008
Last week I got three questions about what I do for a living, and from people who have known me for years. I assume other Postgres and open source people often get the same question; which got me thinking, why is it so hard to explain?
I think I have an answer. Let me list the aspects of my employment:
|o||works in a non-profit field, primarily with volunteers|
|o||helps write computer programs|
|o||works from home|
|o||travels to speak at conferences|
|o||works for a for-profit company|
Now, none of these individually is strange — everyone knows someone who works for a non-profit or volunteers, and perhaps someone who writes programs, or works from home. Certainly some people travel to speak at conferences because you see them speaking at conferences. And working for a for-profit company is very common.
The big problem is putting them all together in a single job; that's just too much for people to process, and if it wasn't me it would be hard for me to comprehend too. For open source folks, it seems pretty normal, but that is only because we have seen it; it is not surprising that the uninitiated find it confusing.
Of course, the secondary question is how does the for-profit company make money in the non-profit field, and the answer is that the company provides support and services for the software, which seems pretty clear, if the person gets that far. ;-)View or Post Comments
Monday, August 11, 2008
I mentioned my son Matthew won a Pogo Linux server at OSCON. Well, it arrived a few days ago — an Intel quad-core, 2.4 GHz server with 4GB of DDR3 memory and a 250 GB hard drive. With the help of Robert Treat during the PG Pool Party, Matthew has installed Postgres on it. Matthew already had a more powerful laptop than me — now he has a desktop more powerful than the server I use for Postgres development too.View or Post Comments
Monday, August 11, 2008
This year's PGDay at Linuxworld was very successful, with fifty attendees. Josh Berkus did a good job of targeting the talks to the audience, focusing on adoption, business issues, and advanced features. The evening party was lots of fun with eight pool tables and free food and drinks. I think we had more attendees at the party than the conference, which perhaps is a good sign. ;-)
Our Postgres booth activity was exceptional — booth visitors were often serious Postgres users, asking for database features, adoption questions, or how to help the project. This is a big departure from the visitors we had only a few years ago. Josh wisely printed a MySQL/Postgres comparison sheet for the few visitors who needed such information.
We did have a few folks stopping by just looking for swag. I jokingly suggested putting up a "No Swag Here" banner.
Speaking of swag, one of the classiest swag items I ever got was an Ingres mug I got from Open CeBIT Australia in 2007. I noticed the Ingres booth had them on display and admired the design and style. I struck up a discussion with Ingres staff about databases, and they gladly gave me a mug when I asked. I think many swag distributors don't expend sufficient effort when creating swag. Postgres swag usually shows good design and attention to detail.
I should also mention I never understood the black shirt/geek thing — black shirts are too hot to wear here in the summer; I avoid them.View or Post Comments
Friday, August 1, 2008
I have been named as a US PostgreSQL Association board member. The original plan was for me to run for a board seat in the next few months but it seems the board wanted me appointed now because Selena Decklemann resigned. I don't know all the details of why Selena resigned but I hope it doesn't affect her involvement in Postgres user groups — she has been invaluable in that regard.View or Post Comments
Thursday, July 31, 2008
This year's OSCON was simply great. There were many excellent tutorials, sessions, and keynotes; far more than in previous years, in my opinion. (I have sent O'Reilly organizers a congratulatory email.) I have three talks ready to submit for next year.
This year I did not spend much time in booths but attended the conference in full, even attending evening parties. The Sun/MySQL party was particularly fun because it was in an empty parking garage with a carnival theme. In the center was a sumo wrestling mat with sumo suits for the combatants. Monty Widenius (MySQL) and Josh Berkus (PostgreSQL) eventually squared off and a video captured the carnage; Josh lost 5-0. (Can we pretend Josh represented Sun instead of Postgres in the match?)
I spent enjoyable time with MySQL's Monty Widenius and Brian Aker at the conference. Monty even brought black vodka to the Postgres BOF to share with us; that was a great gesture by Monty. I even tried a sip and it tasted good, like licorice.
You might remember the OSCON Expo Hall Passport; people got stickers from exhibitors to try to win a prize. Well, it turns out my son Matthew won the grand prize of a Pogo Linux Verona 550 workstation.
I am psyched for OSCON 2009.View or Post Comments
Wednesday, July 30, 2008
I am currently doing a Russian Q&A session hosted by Nikolay Samokhvalov. Oleg Bartunov, Teodor Sigaev, Marko Kreen, and many other Russians are taking part. They have public and private Jabber channels and a Google document for questions. When someone asks a question, it is numbered and placed in the Google document, and translated into English. I answer on the private channel and it is translated into Russian and posted to the public list.
Two hours into the session, we have 25 people who have asked 27 questions; there is one hour remaining. I think this is something other language-specific groups should consider; I would be glad to participate.View or Post Comments
Tuesday, July 29, 2008
The MySQL ecosystem has seen a lot of activity in the past seven months:
|o||Sun purchases MySQL|
|o||Falcon storage engine creator Jim Starkey|
|o||New storage engine fork Maria by|
|Michael "Monty" Widenius|
|o||New database fork Drizzle by|
That's a plot line worthy of a soap opera. Many people like soap operas, but I am not sure it is a good thing for a database. ;-) I wonder if the next seven months will show similar activity.
The new database fork, Drizzle, is perhaps the most interesting because it has the backing of many of the senior MySQL developers. By removing features from the MySQL 5.X code base, it shows MySQL/Drizzle moving away from the traditional Postgres OLTP and data warehousing workload and more toward a web-like workload focused on fast, simple queries. It will be interesting to see how this affects Sun's push to make MySQL 5.X more enterprise-ready.View or Post Comments
Tuesday, July 22, 2008
Today I attended a great Gimp tutorial. The graphic nature of Gimp is best learned in a tutorial rather than by reading a book. I see many good OSCON tutorials and sessions this year -- many more than the last time I attended OSCON in 2005. I plan to attend many sessions this year, rather than spend all my time helping in a booth.View or Post Comments
Monday, June 30, 2008
My son Matthew, 15, is attending OSCON this year, but not to be with me, but rather to attend the conference tutorials and sessions — imagine that.View or Post Comments
Monday, June 30, 2008
I am having a pool party and barbecue at my house on Saturday, August 9, for Postgres community members in the Philadelphia area and nearby states. Families are welcome. Food and drink will be provided, and, of course, swimming is encouraged. Please come anytime between 2pm and 7pm (directions).
In a stroke of good luck, Tatsuo Ishii will be attending the party, so in his honor I have renamed it the "pgpool PG Pool Party". (Last year the event was informally known as "Momjicon".) If you are thinking of attending, please email firstname.lastname@example.org.View or Post Comments
Tuesday, June 24, 2008
I am speaking at two new venues; one is tonight in New York City, the other is in Massachusetts in July; see my events page for more details.View or Post Comments
Wednesday, June 18, 2008
Remember the Soviet Union? It dissolved in only a few years, with no war or revolt. Fundamentally, it dissolved because citizens lost confidence in their economic and political system; the system was built on a house of cards, which explains its rapid decline.
Database software can suffer rapid decline too, especially if users lose confidence in the software. Both Informix and Sybase rapidly lost market share after poorly received releases.
Open source databases are not immune to the problem of lost confidence. Postgres has been able to build on its reputation for technical excellence for many years, and hopefully that will continue; other open source databases might not be so lucky. My point is that it takes only a few serious mistakes to lose a dominant position in the database market.View or Post Comments
Monday, June 16, 2008
This amazing video visually shows the development of Postgres from 1996 to 2006 using Code Swarm. What is really interesting is seeing new names appear and move into the center. My name appears more than it should because I often commit patches from other people.
Update: Video of the last two months of activity 2011-10-13View or Post Comments
Saturday, June 7, 2008
I have attended or expect to attend seven dedicated Postgres conferences this year, in Japan, San Francisco, Oregon, Ottawa, Maryland, Brazil, and Italy; we have never had so much conference activity.
I would love to find a way to illustrate our growth, but I can't. We don't have download numbers because we use so many download mirror sites. Looking at email activity on the "general" email list doesn't show growth. I looked at Goole Trends but saw a decline; MySQL shows a decline too so the decline itself is not significant.View or Post Comments
Sunday, June 1, 2008
I am back to traveling once a month. I have taken a trip once every month since November, 2007, and have events scheduled every month through October, 2008. I last had this heavy travel schedule in 2003/2004. The two reasons for the increase are the greater number of Postgres-only conferences, and the desire by EnterpriseDB to be more involved and supportive of the Postgres community.View or Post Comments
Friday, May 30, 2008
The Postgres core team has announced that there is now a goal of integrating log-shipping master/slave replication into the core Postgres database. (Previously all replication was done as an add-on.) The first goal is to allow transaction log shipping as part of each transaction commit; later, slaves should be able to execute read-only queries.View or Post Comments
Saturday, May 24, 2008
During PGCon, I attended a Covertity talk about the security scanning software they use to scan the Postgres source code and other open source projects; seems they are scanning the Postgres CVS HEAD every day. I am going to work with the tool to see if I can reduce the number of false warnings so we can check the tool as part of our release process.View or Post Comments
Friday, May 23, 2008
PGCon was great this year. There were a lot of new serious users, and lots of people I knew but haven't seen in months. I wish I had more time to talk to everyone longer.
There were three talks that really impressed me.
First was "HOT Inside". The talk was great because:
|o||It clearly explained a very complex feature added in Postgres 8.3.|
|o||It showed the ability of the community to solve very complex problems.|
|o||It used Powerpoint animation in a useful way (a rare accomplishment).|
The second impressive talk showed an interesting use of Postgres. It was about HTSQL, which allows database queries using a URL-style syntax that is designed to be easy for inexperienced database users. I think it highlights the increased productivity possible when you tailor the API to the job, rather than using the same API everyone else is using.
The third impressive talk was quite challenging because it highlighted the complexity of doing in-place Postgres upgrades between major versions.View or Post Comments
Friday, May 23, 2008View or Post Comments
Friday, May 23, 2008
For me, the big joke at this PGCon conference is, "What does Momjian do?". My boss, Denis Lussier, told a group that he thought I spend most of my time on community work because I certainly wasn't spending a lot of time helping the company. Community members said they thought I was helping my company a lot because I wasn't doing enough community work. Seems I have been found out. ;-)
Someone suggested my gift is getting others to do my work, like Tom Sawyer getting others to paint the fence for him. Looking at all the people I have talked into doing things for me, it is hard to argue. Seems there is no word in English for this skill, or at least not a positive one. ;-)View or Post Comments
Thursday, May 22, 2008
The second commit fest for 8.4 is done. It went much smoother than the first commit fest, partly because it was smaller and better organized. Hopefully commit fests will allow us to be much more orderly about applying patches in the future; this is now necessary because the volume of patches has risen dramatically during the past few years.View or Post Comments
Wednesday, May 21, 2008
This year, twenty-six major Postgres developers had a seven-hour meeting at a hotel during PGCon (images). Almost everyone attended, which is amazing considering the travel distances and that many developers are volunteers; this shows the dedication of our community members. The meeting was very fruitful, with many useful discussions and few disagreements. It was very helpful to get everyone together to talk in person.View or Post Comments
Tuesday, May 20, 2008View or Post Comments
Wednesday, May 7, 2008
The May commit-fest is going well, with lots of wiki and committer activity.View or Post Comments
Tuesday, April 22, 2008
We finished our first 8.4 commit-fest a few weeks ago, and it was a painful process. The commit-fest represented ten months of outstanding patches and feature ideas.
Moving forward, the new idea is to track items using a wiki that can be updated by anyone. More importantly, it looks like our developers are getting back into the habit of reviewing patches so many patches are being dealt with _before_ the next commit-fest in May.View or Post Comments
Saturday, March 8, 2008
We have started our first commit-fest. At the start of every major release we have a time where we decide on open items and patches from the end of our previous release, so this is really no different from what we have done in the past. We do have the new capability of adding comments to patch queue emails; comments allow use to communicate more easily about specific items.View or Post Comments
Tuesday, February 12, 2008
We have branched CVS to start Postgres 8.4 development. It looks like we will be in development until November, 2008. We plan to have commit fests every few months so we don't keep patches unapplied for too long.View or Post Comments
Monday, January 28, 2008
I returned on Friday from London and after a weekend relaxing I am starting to read the one thousand emails I accumulated during my trip. Strangely when I am away from Postgres email for a while I miss it and look forward to returning to it. (Is there something wrong with me?)
In London I did three days of Postgres training for EnterpriseDB, and then a London Postgres community meeting (mostly about Postgres 8.3) with thirteen people. There is talk of starting a regular UK users group.
I did a little sightseeing in London and spent most of my time at the British Museum, which was two blocks from the EnterpriseDB London office.View or Post Comments
Saturday, January 26, 2008
Contrary to previous statements I have made, I will now be attending FOSDEM '08 in Brussels, Belgium. FOSDEM seems like it is going to be a must-attend event for Postgres users in Europe. I attended FOSDEM in 2003 and was very impressed.View or Post Comments
Thursday, January 10, 2008
I have read sections of the Postgres manual when updating the text, but never read through it. I thought this would be a good time to do that. I am almost done chapter 8 (data types) and have found the need for only small wording improvements.
It is amazing how good the manual is, especially considering how poor it was when I started in 1996. Tom Lane felt the same way when I talked to him about this. Peter Eisentraut did a lot to improve it. He made massive documentation changes when he was translating the documentation to German.
I sent the entire 2000 page PDF document to Staples for printing. It was printed as seven perfect-bound books. It only cost USD $100. There is no way I would have been able to read that text much from a screen.
I am not sure I will finish in time to get my changes into the Postgres 8.3 final release documentation.View or Post Comments
Thursday, January 10, 2008
People are always asking for benchmarks of Postgres against other databases and between Postgres versions. We usually avoid citing specific numbers because different workloads show different behaviors. However, this benchmark is very helpful because it shows pgbench (like TPC-B) against Postgres versions 7.3.X to 8.3.X using the same 8.3 version of pgbench:
Version 7.3.15 7.4.13 8.0.8 8.1.4 8.2.beta1 8.3beta1 tps 311 340 334 398 423 585
While the comparison isn't going to be useful for all workloads, it does show how far Postgres has improved in performance.View or Post Comments
Thursday, January 10, 2008
Looks like we will need an 8.3RC2.View or Post Comments
Thursday, January 10, 2008View or Post Comments