(L): Mid-1990s during summer in North Branford, CT; (M): October 2002 snow and colorful foliage at PSU; (R): Photographs and temperatures during a 2017 solar eclipse
The (marginally) sordid but (definitely) interesting history of the coolwx.com web pages.
Given some of these pages are now nearly 30 years old [a lifetime in this medium], and given the unique conditions that led to their birth and growth, it is worthwhile to give some history on them. Since I receive emails from users every week graciously commenting on the value of the web pages to them, and a few occasionally asking about their history, it's time to document the (somewhat) sordid details.
1991-1992
During my first year or two as an undergraduate at Penn State, I was being introduced to all the typical things a new meteorology student would in terms of classes. However, what was quickly evident was that access to real-time meteorological data was frustratingly difficult. Very frustrating. While the internet existed, it was in such a embryonic state that few had access and no one made available weather data online. Literally the day I was moving into my dorm at PSU in August 1991, Hurricane Bob was making landfall in southeast New England impacting the area in which I grew up - - and I had no sources of weather data [not even TWC, as there was no TV in my dorm]. A year later, during the beginning of my sophomore year, things had not improved. Hurricane Andrew was making landfall in Florida and the ability to look at observations was basically nonexistent without paying a company across town for their own interface.
While there were software programs within the Department of Meteorology that connected to NOAA data feeds, the VMS/VAX system was not accessible then from a dorm room or apartment, simply because technology didn't allow it. Later that year, I recall vividly on December 11, 1992, walking across campus from my dorm to Walker building in horrendous slush to get access to data to see what was going on across the East Coast [a rather intense coastal storm] as I grew up and had family in Connecticut. Rain was changing to (an unforecast) blinding snow across southern New England, and the subway system and some highways in NYC were going underwater from storm surge. I tried my best to locate and read the buoys just off the coast for insight into the evolution of the storm, but the data was encoded and took too much time to decode manually.
1993
During 1993, I got access to a few Unix systems at the University -- partly through the aid of friends who worked directly with those groups. Combined with a heavy dose of curiosity, a larger dose of determination, and some filesystem browsing, I began to slowly learn the art of scripting in Unix by example. [There was no Google or Wiki from which to learn this stuff, and I certainly wouldn't RTFM]. Some of the initial searching and coding took place on a Commodore 64 running kermit to telnet to the Unix accounts on campus. The editor? vi, of course. Some months later - I think once I found Charley Kline's wonderful and prescient wxmap program and its interface -- I found links to the first online weather data sources. I believe the interface would have been ftp or gopher at best. The real-time access to raw surface observations, raw buoy and ship data, was a new world, and overwhelming -- but exactly what I wanted. In April 1993, Mosaic was released, and although quite kludgy, simplified the browsing of remote directories for those observations.
1994
By 1994, I had learned enough Unix scripting and C, Fortran programming to create solutions to the problems that had dogged me earlier. I can confidently say that my knowledge of scripting and its automation had easily the largest impact on my efficiency as a scientist the remainder of my (not yet over?) career. [onpedestal] If you are a student seeking a meteorology degree (and definitely if a graduate degree), learn both as soon as you can without getting in the way of your coursework [offpedestal]. As the hurricane season got underway, I had wished (again) to be able to watch the buoy and ship reports to track in real-time changes in the hurricanes. With access to some ftp sites, the "expect" scripting language (now replaced most thankfully by wget and curl), and a superficial understanding of cron, I could automate downloading of the raw reports. A few days of coding later and learning-by-example for html, my first "real" web page was born. The web page would decode the horrendous numerically coded buoy/ship/cman observations and make them available in tabular format for me (and the world) to see. The web page was announced on 8 September 1994 on a few weather-related listserv groups, and Google has archived (for better or worse) the announcement:
http://tinyurl.com/29mdjxo
Two months later, Hurricane Gordon approached the NC Coast and I used the web page in real-time to watch the pressure fall, wind and waves rise, as the storm approached:
http://tinyurl.com/2cfjd7l
As someone who is still fascinated by watching real-time observations come in, there was something remarkably satisfying and exhilirating about designing a tool to solve a problem that is then also beneficial to others. More than a few emails came in from students, forecasters (including NHC), and a few in the general population who greatly appreciated the content. I also received an email from Tommy Owens, then a commercial fisherman from New York, who found the web page immensely useful for his job. He later informed me that this and other web pages of mine served as inspiration for him to seek a degree in Meteorology a few years later at PSU. He now has a very successful career at NASA. It is a remarkable feeling to know I had a small part in that, and it continues to drive me today.
Over the course of the following months, several additional features were added to the web site: searching (by buoy/ship ID, by lat/lon box, and by distance from a chosen point), as well as ascii-based charts of the conditions over time for each station. The popularity grew, and it became addictive--if not intoxicating--to watch the web access logs to the content, seeing from where people were accessing the web page (but of course no private information). I suspect I am not alone among web developers in that fascination. During the years following this Offshore Weather Data web page, many of the ideas and tools provided by the web page were emulated at the National Data Buoy Center (NDBC). I'm told imitation is the sincerest form of flattery. Of course, I hear that more often from those that imitate rather than from those that are flattered. NDBC's web page long ago surpassed mine, but there are still a few unique aspects to coolwx.com/buoydata -- such as plotted observatons, contoured maps, and hurricane-centric maps. Oh, and the fact it was first.
A month or two after this first web page, there was growing interest on the sci.geo.meteorology usenet group for a method by which public observations (e.g. yours and mine) of weather could be collected and displayed. The rapid growth of the internet quickly exposed the general public to weather data and many wanted to communicate what they were seeing. It was truly a unique time of rapid change, and I am grateful to have been there at the age I was. A request was made by Adam Gaffin (more recently of Boston Blog and Universal Hub fame) on the newsgroup on Nov. 30, 1994, for such a tool and [armed with my somewhat inflated confidence of web development from the Offshore Weather Data webpage] I naively volunteered to setup such a database and web-based interface:
http://tinyurl.com/ydxm64os
This page required a whole new level of complexity, however, and tested my limits at the time. Any user in the world would need to have the ability to report their weather conditions in a logical format, the web page would have to be able to update the database in real-time as other users browsed it, and (with time) maps plotting the observations would be produced. This wasn't simply a page that updated in the background by crons. It would update as people changed it by submitting their observations. After an almost violent stream of coding and experimenting and learning a few additional languages (awk and perl), and definitely a lack of sleep and more definitely a lack of attention to classes and homework [don't tell], four days later the web site was announced on Dec 4, 1994:
http://tinyurl.com/2fzr46h
Based on my post, it seems Netscape was already available. I had forgotten that. The response to the new web page was immediate and strong. The first posted usenet response to the web page (by Todd Gross of "Perfect Storm" fame) at the same link above, was insightful commentary on the state of the web at the time:
"Bob.. I LOVE your database. [cut]. The problem I see (minor) is that
a.... One must log into this home page, it doesn't come to you
b.....Most do not yet have access to WWW
Let's definitely keep with this though, I think it is great. [cut]
-Todd"
No doubt those who are less than 25 years old cannot fully appreciate point (b) above.
The first submission to the database was, I believe, from New Jersey, only an hour after the web page was announced. Unfortunately, the archive before 1997 was lost, so I cannot be sure. If you are out there and know you were the first, please do drop me an email.
A few days later, as noted on the newgroup replies above, my (now longtime) friend Jeff Wolfe (then PSU College of EMS) offers to host the web page at the College level. The web page remained at that eventual location for nearly 10 years, until my move to Florida State University. Perhaps not ironically, several times over the years observations were submitted by students from the weather station map room at FSU Love Building, the building at which many years later I would find employment as a Professor. Small world, indeed.
Only a few weeks after the website announcement, the infamous "Santa Storm" of December 24, 1994, struck southern New England and Long Island with unforecast storm to hurricane force winds. I recall the flood of observations posted to the web page from weather fanatics (like myself) in the region who were as in awe of the wind [I was home for the holiday]. It was the first real-time demanding test of the web page and it survived the beating.
For a few years the observations were used by local National Weather Services offices (particularly in the Northeast, thanks to Walt Drag) to complement their database of storm reports from cooperative stations. However, the lack of quality control and the tendency for some to intentionally exaggerate reports, such as snow totals, led to the abandonment of this. However, CoCoRaHS was established several years later and I have been told the interface and experience of coolwx.com/userobs played a role, even if small, in that.
Over the years I received email from scientists at McMurdo Station, Antarctica, commenting on how the site keeps them in touch with worldwide weather:
"28 December 2001
We're using your websites to track the course of Icebreakers,
among other things. I notice you've got weather, maps, and
a whole encyclopedia of other meteorological information.
It's a fantastic site, very educational and entertaining for
those rare moments when we have free time to browse web sites...
Joel Foy, McMurdo Station, Antarctica."
A few times those Antarctic scientists even submitted observations to coolwx.com/userobs. Talk about living precariously through the lives of others...
As of late 2023, over 1.5 million observations have been submitted to the database, with 95% of them from the USA. These are volunteers from around the world taking their time to report what they see. That is remarkable to me -- that over 1.3 million times someone went to this web page, entered weather data, and hit "submit". Perhaps more remarkably, several users have written their own code to automate the submission of their Davis Weather Station observations to the web page. There are now maps of the plotted observations across the world and timeseries of individual users observations, thanks to the wonderful software package, GrADS. The coolwx.com/userobs database is searchable by any phrase, and documents major events since 1997. The comments section of the reports permit users to enter any comments they wish [both a strength and, due to the childish, a weakness]. Nonetheless, the comments document often non-meteorological events, such as real-time reporting of the tragic 11 September 2001 attacks, the observations of the Space Shuttle overhead, including the Shuttle Columbia's loss in 2003, and eclipses.
1995
My own obsession with the statistics of meteorology led to the creation of the U.S. Weather Statistics page a year later, which was a much more straightforward page that still drew an audience at a time when the web was mostly devoid of such information. In 1995, Point.com and Magellan.com (now I believe both gone as they were originally known) rated the three web sites as part of the "Top 5% of the Web". I was sent a congratulatory letter in the mail and a request to sign a form giving them permission to include snapshots of the web pages in a book they were producing on those top 5%. I remember seeing the book in a Barnes and Noble a year later. I debated buying it, and now regret not doing so. While it would be much harder to be in the so-called top 5% now, it had more meaning back in 1995 -- as a marker of foresight and substance, given that the web is now dominated by a vast wasteland of increasingly stale cotton candy. Update (20 September 2012): In another illustration of how small the world is, a former FSU student of mine (thank you Jenna!) upon reading this history alerted me to the fact that this book is available at Amazon. It has been purchased (although the shipping was more than the book....) and my library is now complete.
During the summer of 1995, after graduation but before the start of graduate school at PSU, I spent some time at PSU during Arts Festival. I brought over 60 floppy disks with me to download Slackware Linux in one of the PSU Computer Labs [modem from home was not feasible, given 1200 baud rates], which I then installed on my new 100MHz Quantex PC after returning to Connecticut from PSU. Installation took hours and the manual build of the kernel alone took an entire night as I recall. And getting the video cards to be recognized and correctly configured... anyway, I digress. Linux installers/users today have it very easy. So, having Unix locally really helped improve my coding skills.
1996-2002
There were many other pages developed over the years, all of which are now at coolwx.com, but none were as personal as those first three. They represented a remarkable confluence of timing of a desire to see what I was learning in college within real data, to learn Unix, scripting, real programming, and all coincident with the birth of the web--the latter of which provided an audience to share my personal curiosity. Five years earlier and I would not have had the tools to accomplish what I wanted. Five years later, and the web may have already had what I was looking for, and what is now a part of my fundamental existence would have been replaced with something unknowable, and most likely without my contribution.
2003-2005
In 2003 I began my career at FSU Meteorology and moved the web pages. Initially I placed them on my research server at work, the equally popular moe.met.fsu.edu. At first it was difficult to continue maintaining some of the web pages, partly because a large part of the coding needed to be updated for new operating systems - but also because my new job required a greatly decreased dedication to web page development. Grudgingly, I decided to shutdown many of the web pages, and announced as such. The response was striking, as I received dozens of email pleas to keep the web pages going. I even received a mailed letter (now lost in my office) from someone in the Midwest (I think Nebraska or Oklahoma) who rather angrily chastised me and FSU for being so rude. Not until that moment did I realize I had created something that others truly relied upon.
Not yet appreciating the demand of the web pages, I had accidentally left one of my web page's directory structure open, leaving the ability to grab the scripts that I had written to generate that web page. An employee (unnamed) of a private company (also unnamed) grabbed the entire directory structure with the aims of replicating the web page given his/her keen interest in the content. The fact it never happened is, I think, a reflection more upon them than me.
The combination of all of this drama convinced me to keep the web pages going. However, it became clear that the pages deserved their own domain to separate my "real" job from this other stuff and provide it the necessary resources. After some discussion and guidance from my friend Geoff Fox on his experience with hosting companies and his own very successful blog, coolwx.com was registered on August 16, 2004, and the pages were moved there soon after. The hosting costs hostforweb.com were initially cheap, but as bandwidth and graphics creation loads grew, the hosting costs grew quickly. To compensate, I moved to a dedicated rather than shared server,and also added ads on the web pages. While this latter decision remains a conflict for me - as I realize the ads are distracting and as someone recently said in a mostly movie, "not cool" - I hope that they are far less distracting and invasive than many other weather-related web sites. Update (2 December 2018): In the past year my chosen host provider switched to hostwinds.com for very valid reasons well known to other customers of the prior host provider mentioned above. Further, ads have been removed from coolwx.com as it seems only appropriate at this point in my career that the page should stand on its own, and less cluttered by those ads, given the 30-year anniversary now approaching. I thank everyone who has helped support it either through clicking on ads or the donation link previously on there over the past 14 years.
2006
When all was finally stable with the web pages, just before the start of the 2006 hurricane season, scandal hit:
http://tinyurl.com/2cefx2c
Some groundbreaking meteorological research at FSU (by another faculty member) was privately funded with restrictions on dissemination. That research produced a very valuable forecast tool. Due to a combination of misunderstanding and ignorance, FSU demanded that no real-time weather data or forecasts potentially in any way related to hurricanes be disseminated by FSU faculty or students. Talk about a large net? Since the scope of the demand was not explicitly limited to FSU web sites, it was assumed this applied to all relevant web sites run by FSU faculty or students. All content on coolwx.com [and moe.met.fsu.edu] was suspended for a very long two days. Before the misunderstanding and overreaction was resolved, I received hundreds of emails, the Administration of FSU received dozens of angry voicemails and far more emails, and a shiver down the spine of the collective meteorology academic community was felt and alleviated. A potential dangerous precedent didn't happen, but it was close enough to leave a very bad taste that can't ever be forgotten.
2009
The pirates take notice of coolwx.com...
I came back from a conference trip to find anxious phone calls on my work phone, home phone, and some equally anxious emails. Apparently maps from coolwx.com/buoydata and other offshore-data related web pages (e.g. oceanweather.com) were found printed out on pirate ships that had been seized in the Indian Ocean. Pirates were using the ship weather reports to track the ships' locations and intercept them. A nauseous feeling overtook me -- the feeling resulting from naively creating something good that was used for something rather horrible. As the phone calls and emails revealed, NOAA was requested by the US Maritime Department (MARAD) of the Dept. of Transportation and Dept. of Homeland Security to place a significant delay on the display of ship observations in the region of the Horn of Africa and the Malacca Straits. Ethically agreeing of course, but also not wanting people in black suits and sunglasses to arrive at my office, I agreed to not plot ship observations in this region of active piracy. While the decision was a no brainer for me, I lament that my proposed (and more creative) solution wasn't accepted -- altering the latitudes and longitudes of the threatened ships on the web pages to guide pirates toward undoubtedly well-armed United States Naval vessels...
"Today"
While at FSU, a few additional web pages have been added to coolwx.com. All have the theme of providing unique information about the weather that is difficult if not impossible to find elsewhere. Once any of the web pages is no longer unique, it will no longer be maintained. There is no value to me in replicating what others have done on the web. That the web pages are still heavily used after nearly 30 years is amazing. And I still look at them every day, for my own curiosity (after all, that is why I created them in the first place). I occasionally use them in my courses (in particular, Synoptic) when they are relevant. When I see links to the coolwx.com images on discussion groups, blogs, National Weather Service discussions, and occasionally even on TV (local TV, CNN, and a few times TWC), I grin a bit at the fact that it all started 30 years ago from my Atherton Hall PSU dorm room on a Commodore 64 using telnet.
I still use only vi for editting code. And I still have the same Commodore 64 -- although it collects dust these days in my office at work garage at home.
Although -- one day in 2005 my graduate students and I started it up, and we were able to load (via the 5" floppy) and briefly play Video Poker. After a few hands, the smell of burnt dust was followed by gibberish on the screen and the wisdom to quickly shut down.
Thanks to many over the years for their support and feedback, including but not limited to: Jeff Wolfe, Rich Grumm, Jack Beven, Mike Pontrelli, Geoff Fox, Walt Drag, Brett Albertson, Ben Karhan, Rob Brewer, Tim Robinson, Bill Fenner, Greg Forbes, Alistair Fraser, Jeff Buechler, David Roth, and the fantastic team of GrADS developers at and beyond COLA/IGES including Brian Doty, Jennifer Adams, Mike Fiorino.
Bob Hart (rhart@fsu.edu)
First Written: 8 September 2010 [on the 16th Anniversary of the first eventual coolwx.com web page]
Last Updated: 19 November 2023 for several important updates.
