The Challenges of Open Government

A few posts ago, I described the Chief Information Officer positions being implemented by a pair of cities and the implications for the open government movement.  I also made the connection between urbanism and open government.

While researching the post, I spoke and exchanged emails with Dan Lyke, a former member of the City of Petaluma Technological Advisory Committee.  Lyke provided thoughtful and insightful comments about the potential for and challenges of open government.  The comments deserved their own post.  Below, with Lyke’s concurrence and final approval, I’ve reformatted and slightly edited his email thoughts.  His cooperation is much appreciated.

"I think that most people in Petaluma City Hall would be glad to make information more available.  However, there are three concerns:

“First, some of the information isn't available in electronic form. There are a whole lot of current processes that take paper forms in and output paper forms.  There's no point in the process where the data is in electronic form.  Nor is computerizing processes always the right idea.  Computerizing processes is hard and can be expensive.  Also, it's difficult to duplicate the flexibility of paper forms, with notes on the margins and additional comments stapled on and such.

“Second, a lot of the data that is available in electronic form isn't really useful.  Take the budget documents, for instance: They're pdfs with formats that change every year and have no semantic information attached to them.  If you want to figure out what the spending trends are for public safety, for example, you have to go back and find the appropriate table in each one and copy and paste that data into a spreadsheet, and make sure you're comparing apples to oranges, etc.  In that "human readable" form, they're not terribly useful.  Hang on to that thought because I'm going to come back to it in a second.

“Third, when the information is available in electronic form, there's still a cost to make sure that the data as published is current, that it’s relevant to actual city processes, and that it doesn't cause extra support load in City Hall.

“The second and third points lead to the more difficult issue: Data can be good enough for a process, and still be very flawed or poorly understood.  Throwing that data out on to citizens who don't understand the processes that generate the data can result in a lot more work for City staff and a bunch of political maneuvering and hassle for elected officials.

“Two off-the-top-of-my-head examples:

“First, to the city budget issues, I once tried to do some comparisons of our public safety expenditures relative to other places, so I pulled up our budget side-by-side with Novato's.  Similar
population, similar geography, but their fire department had a budget that was something like ten million bucks more than ours. Turns out their fire department also covers some portion of the county, and has a different payment structure because of that, but you could easily see someone taking an imbalance the other way and writing an Argus Courier editorial about wasteful spending and starting all sorts of unnecessary noise.

“Second, I got a copy of the GIS data from City Hall (I think it's all online now), and downloaded
everything the County has online, loaded them up into QGIS, and discovered that some of the lot lines don't match between the county and city data. This really isn't an issue because mapping is really hard and surveying happens between hard markers, not based on GPS (for one thing, land, especially here in California, moves), but you could see how a mismatch like that could throw neighbors into tizzies over fence lines and who's squatting on whose driveway and...

“If you talk to City Hall about making data public and trying to open up the processes of government, you will find a lot of excitement and encouragement.  I've been told that much of City Hall leadership is very open to the idea, with some actively looking for opportunities to collaborate with outside developers.

“You'll also find a little bit of caution and a whole lot of concern about additional costs.

“However, this also leads to the other side.  I've been out there pushing the notion of ‘let's look at this data’ to everyone I talk to, and have been hitting a lack of interest. Before the city upgraded Granicus to allow for the current podcasts and variety of formats, we had citizens come talk to the Technology Advisory Committee about making the video and audio from the town meetings available in more open formats.  I set up a system to do that, emailed the people who spoke, and heard nothing.

“There are a number of people who are talking loudly, but are more interested in issue awareness than solutions.  That's fine, but I'm a solution-oriented person.

“Finally, this is exactly why Jaimey Walking-Bear and I set up PetalumaOpen.com.  So far it's just been me and Mark Petrovic trying to put together things about the city we find interesting, but there are things we can do there that the city can't.  For instance, what if the City has a document that's a scanned pdf but you want a text version?  The city can't really OCR (Optical Character Read) it because they've got to support data quality. But Jaimey and I can run it through an automated OCR process and re-publish it.  I can then absorb the support issues.”

Thanks to Dan for his comments.  I can add a couple of further data insights from my career.  First, the assessor mapping issue he noted is very real.  Final maps for subdivisions are calculated, mapped, and field monumented to the nearest 0.01 feet.  In a community where I formerly worked, the assessor’s office took our final maps and had an intern scale the data onto their maps with a writing implement that drew a line that scaled to a half-foot in width.  I’m not making fun of the interns.  They actually did a credible job given their tools.  But the accuracy was far below the recorded maps.

When GIS came along, it was the assessor’s maps which were digitized, while the original maps continued to hang untouched in the County Surveyor’s office.  (GIS budgets rarely have the resources to process the more detailed maps.)  And the first source when property owners had questions about parcel lines was the GIS mapping.  You can imagine the confusion.  In the engineering/surveying world, GIS is often described as an acronym for “Get It Surveyed”.

And the issues with as-builts are even worse.  As-builts are the supposed records of where improvements were actually built.  They’re particularly important for underground improvements.  But for reasons pertaining to how the construction process works, there is a large possibility of errors.  And the further we go back in time, the more the uncertainty grows.  City engineers are justifiably twitchy about as-builts that date from the time of their great-grandparents.  But that’s a story for another time.

Thanks again to Dan Lyke for sharing his insights.

As always, your questions or comments will be appreciated.  Please comment below or email me.  And thanks for reading. - Dave Alden (davealden53@comcast.net)