PubForge Blog

(This post began life as an email thread, but maybe needs to be more public so here it is.  Edited and expanded for obsessive clarity…)

It strikes me as somewhat simple (OK maybe not exactly simple) to develop a Drupal-based CMS with enough commonly-needed features for public radio/TV stations.  You’d have your pre-built data types, skinable templates, forms, and possibly a set of pre-defined roles and workflows.  All nicely documented etc.

But what we all really want is a system that knows about media files.  You could upload (or link to) a media object, and the CMS would extract its available metadata.  The system would then save that metadata in its database for processing and display in various ways.  On web pages where media is published, the system would display its media type, length, bitrate, framerate, whatever.  Then of course we’d be adding by hand other metadata like title, subject, author, keywords, description, etc as we add media content to the website.  Ideally, the system would be able to automatically read ID3 tags, MXF, and EXIF metadata for both technical and descriptive information.  The idea is to automate the capturing of metadata as much as possible.

For web pages, we’d probably want to display mostly descriptive metadata, and not things like sampling rate, bit depth, color format, etc.

But for RSS feeds we need some of that technical metadata like filesize and mimetype.

And here’s the good part: If we capture enough information about our media objects, we can easily express it as “shareable metadata” via PBCore-compliant XML, and other standard schema.  So the CMS becomes a powerful tool for creating a large index of public media.  We can then write applications to search that index at a very fine level of detail.  Think Technorarti only focused entirely on media objects expressed as detailed XML records.

At WILL we currently catalog media objects (as I call them) using our CMS, but there’s no automatic extraction of anything.  We have to key in all the data.  But once that’s done, the output looks like this:

http://will.illinois.edu/metadata/pbcore/pf2008-04-17-a

Seems to me this is the beginning of a system-wide super API that doesn’t depend on any central organization, and is truly open source.

Existing open source PHP functions for automated metadata extraction could be integrated in a Drupal-like CMS.  The PHP ID3 function allows for reading and manipulating ID3 tags; the PHP Exif Functions can extract all kinds of metadata from JPEGs and TIFFs.  Similar functions may already exist for video files.

If we have a CMS that understands how to read existing metadata from the digital objects we feed it, we’re half-way to building an online digital asset management system.  More on that in Part Two…

Jack Brighton

Under the category of be careful what you ask for…we created a survey to determine how public radio/TV stations are maintaining their websites, their web staffing levels, technologies, and needs. We wanted to find out about the use of open source tools, and maybe identify development projects that would help people throughout the system.

The survey remains open, and we hope lots more people fill it out.

Meanwhile, here’s a quick summary of the stuff that really caught our notice. Made our hair stand on end. One of us cried. Here’s the scoop:

1) We have 64 survey responses from public TV/radio stations as of September 4, 2008.
2) 32 of those stations have no full-time staff dedicated to website creation and management.

Ouch.

3) 18 stations have at least one part-time web staff; 10 have two part-timers.
4) Only 15 stations have a single full-time staff devoted to web development. So that’s what? Less than 25 percent of the respondents.

That’s crazy.

5) Eight responding stations have two full-time web staff; four respondents have three to five full-time web staff.

The picture thus far would include about 28 stations with one or more full-time web staff, and 32 stations with no full-time web staff. (The numbers are a bit odd due to the survey design, but that’s an accurate-enough count.) Some of the 32 stations don’t even have part-time web staff. I hate to go out on a limb here, but I think that reveals a significant deficit in web staff resources at half the stations responding. But wait, there’s more.

6) Only 37 respondents out of 64 have staff skilled in web design, and only 29 have skills in web standards/CSS.

We’ll resist commenting on the difference between skills in web design versus web standards. The numbers in either case are way too low.

Look at it like this: If your station operates a transmitter, odds are better than 50 percent that you have an engineer or three who knows how to make the signal go from microphones and cameras through mixers and switchers and out to the air so people can receive it. Metaphorically speaking, a web server is like a transmitter but rather than mics and cameras we have websites and RSS feeds. Our station websites are now mission-critical: they play an increasing role in delivering content, engaging audiences, fundraising, and delivering on our promise to enlighten communities and inform democracy. So far this survey shows that at least half of us don’t have a single web engineer.

So NPR has rolled out a wonderful API, and soon social networking features and tools. Those of us who understand how to use an API etc can take advantage of these cool new things and make our sites even better. The problem is, too many of us don’t have that understanding.

But let’s not take that as a criticism of stations without adequate resources and skills. In building public media on the web, we’re in this together. That’s why we began this assessment: to find key areas where we can collaborate, share knowledge, and develop best practices.

It’s just that the contrast between the haves and the have-nots is so glaring, and it seems relevant to highlight for further discussion. After all the CPB charter includes the following:

“The mission of CPB is to facilitate the development of, and ensure universal access to, non-commercial high-quality programming and telecommunications services…
…Whether through direct services such as assistance in management practices and access to new technologies…”

Points of Community Leverage

The survey also shows wide use of and interest in open source solutions, and a desire for collaboration. A growing number of stations are using Content Management Systems, including Drupal, Zope (OK it’s really a framework), Joomla, ExpressionEngine, and Public Media Manager. Others are using home-grown CMSs built with open source elements and scripting languages. Open source tools and systems are used by almost half (31) of the respondents.

21 of the respondents expressed concerns about using open source based on…their lack of staff skills and support resources. The same barrier rears its unlucky head once again.

The good news is the vast majority of respondents are interested in learning more, and collaborating around open source projects that would serve the needs of all stations. A solid majority of respondents say they would travel for training sessions.

You can of course read the full analysis of the survey on this site, and we’ll update it as we get additional responses. Meanwhile, it looks like PubForge has an opportunity for some heavy lifting. We’d like to make a big difference for both the haves and the have-nots. And you know what they say about many hands making light work. Let’s use this as inspiration and get busy.

As I’ve been reading over the PubForge Open Source Collaboration Survey results (over and over again), I’ve begun to see between the lines a bit. I don’t mean I’m getting dizzy, although that could be true as well. No, I mean that as is often the case, I find myself beginning to understand the hidden meaning and unspoken concerns that always seem to poke their heads up gradually as you review data over time.

The head that I’m seeing poking it’s head up at the moment has a lot to do with sustainability for public media web operations. We talk a lot about this subject at conferences and on webinars. But I wonder, as we have those discussions, have we really given thought to what it means and why we should be concerned. It seems to me we can’t sustain ourselves by trying to figure out how to do the web better than we are doing it now.

Let me get right to the point: Sustainability does not mean putting more time and money into the web or web projects or figuring out how things work on the web. It means putting more time and money into the people who run the web and web projects, so they can develop policies, systems and processes that will make it possible for our projects to sustain themselves.

Two simple facts pointed me in this direction of thought.

First, that the single loudest cry for help found in our survey results was the repeated, “heellp meee…” that resounded when we asked everyone, “what do you want to do and why haven’t you done it yet?” We saw a lot of answers along the lines of “no tech staff” and “no support” and “limited expertise”. In short, we’d love to embark upon the greatest web project of all time but we’ve no idea how to do it or how to keep it going over time.

Second, the number of great ideas that came in through this survey was amazing. So far we’ve seen an amazing outpouring of what amounts to contributions to collaborative projects, and we haven’t even created a mechanism for doing so yet. There are a lot of great people out there in public media who have all kinds of ideas about how to make our services better, richer and more competitive.

I take these two facts and I add them to something else I know about having been a part of the Public Media Metrics project for a while now: Most station websites are lucky if they see the same visitor come back to their website more than once each month. Repeat traffic rates for public media station websites is pretty abysmal, and this suggests that creating any kind of model for ongoing web presence sustainability is laughable.

What’s my result after putting it all together? Let’s see…hmmm… for the geek audience here, I think it’s something like this:


$sustainable     = false; // ASSUME THE WORST AT THE START AND TEST IT OUT LATER
$good_ideas      = 1000000;
$visitor_loyalty = count($visits_per_visitor[$month]);
$have_help       = count($web_staff);
$have_support    = getSupportFromVisitors($visitor_loyalty);

foreach ($good_ideas) {


if ($have_help == true && $have_support > $not_much) { // $not_much IS NOT DEFINED

$sustainable++; // BONUS!!!

}
else $sustainable--; // BOGUS!!!

} // LOTS OF THINGS STAND IN THE WAY OF GETTING HERE...

Sorry - for those who don’t eat code for breakfast, this mostly breaks down to, “if we don’t have support from visitors, and we don’t have staff, we can have a million good ideas and it still don’t amount to spit.”


// === END SARCASM === //


One of our survey respondents made the point that we should ask CPB for help. At first glance I thought, CPB has helped quite a bit, so why are we still here? Is it because we don’t know how to make good things happen when we get money? Is it because we are wasting it when we get it? I don’t think either one of these things is true. In fact, I think that there are lots of folks out there in public media who have done a lot of good things online with the money that’s been granted by CPB and other organizations. We are a productive and thrifty bunch.

It seems to me that the real issue when it comes to sustainability is all about infrastructure. It’s all well and good to grant $10,000 (or way more) to spend on the coolest web project that ever was. But how many stations would have the expertise and services they need to do anything with that kind of money? There are many, many stations out there who would love to have that kind of a budget for use online. But I’ll bet they’d like to spend it on salaries, not on “projects”. I may be shooting myself in the foot here as I am a consultant myself, but what good does it do to send a heap of dough to a station to spend on hiring consultants who will build a service or feature for them that, once those consultants are done and gone, the station won’t know what to do with over time? Should that station hire the consultants again to come back and fix it/use it/upgrade it/update it/expand it… you get my drift.

We wouldn’t put a station on the air without having programs to air, a tower to transmit, and staff to mind the programs and the tower that sends it out, BOTH. Likewise, we cannot expect to put a quality online service into the ether without having content, a website, and staff to mind the content and the website that sends it out, BOTH.

PubForge was formed of this notion that we are under-resourced and that together we can help each other with problems that would normally be insurmountable on our own. The very concept of PubForge is to build a community of virtual staff who can materialize out of thin air when the need arises, for whatever station might be in need at any given time. But the one thing that PubForge can’t do is provide sustainability for each individual station. All the great collective ideas and collaborative systems and community projects in the world will not change the fact that without a human being or two sitting at the keyboard punching code or publishing content, there is no way to sustain the web operation.

So what is necessary for sustainability? We absolutely MUST invest in more than just projects. We need at least one, if not two full time bodies in each station that hopes to have a web service. Granted, once those bodies are in place they’ve got their work cut out for them. There’s no shortage of problems to solve for the full time web pro in public media, starting with building visitor loyalty and reaching all the way to monetization. Those are huge ailments in need of serious triage, and any emergency team attempting to heal that patient will be pretty busy for a while. Thing is, what would you do if you had that patient right there in your station, but no emergency team attending?

I can think of at least 33 stations right off the top of my head who don’t even need to guess at the answer.

What have we done with the new NPR API? This would be a good place for people to share examples of cool mashups and apps they’ve devised to tap NPR’s open content. Or to suggest ideas on which we could perhaps collaborate.

Here’s one of mine: What if I tagged my news stories, interviews, and other content with keywords based on the NPR taxonomy (or even just my own keywords) and when I publish that content on my website, it generates a query to the NPR API? I could have a widget sitting next to my content pulling in related stuff dynamically. As the NPR API expands its reach to other public media sources, each content entry on my site becomes an entry point to a growing universe of related content. Of course that universe might get pretty big, so what if we wrote a script that could then parse everything and generate a navigational structure based on the metadata returned with the results of the query. So the query results would evolve over time, and so would the navigational structure.

What if next we expand the range of sources to query based on the metadata of our initial content? Scientific and cultural institutions have large collections of content, many accessible through an API. Funding is increasingly premised on open collections and public research results. What if we tap into that, so a given media object can serve semantically and programatically as merely a starting point to explore a growing web of deep knowledge and perspective?

Maybe that kind of language sounds cheesy or something, but it seems like a fun thing to me.  On the other hand, maybe let’s start with the NPR API and go from there…

Jack Brighton
WILL Public Media

Those of us consumed with passion about metadata for A/V objects (and who isn’t…) have been excited by the emergence of the PBCore. We present here an update.

In our last dramatic PBCore episode, CPB funded a multi-year project to develop a standard for shareable metadata about audio and video productions and files. This culminated in the release of the PBCore Data Dictionary and an associated XML schema, with Version 1.0 in April 2005, and an improved Version 1.1 in January 2007.

We’ll leave an actual description of PBCore for another time and place, or get full details on the PBCore site.

It turns out PBCore is darn useful. Film archives, academic media collections, and media curators including the Library of Congress are actively pursuing systems that speak PBCore. Not to mention PBS, NPR, and a growing number of local stations. At recent AMIA conferences PBCore has been a central topic. PBCore has become relevant and possibly important to all moving image archivists, because it fills a black hole in the metadata universe concerning digital media.

Color us surprised when the initial CPB project to develop and support PBCore ran out of money last August. Forthwith, the principle developers at WGBH and elsewhere proposed a second phase, to establish a PBCore change-management process, plus funding to maintain the website, workshops, and other support activities. So far the response from CPB has been PBCore who?

What’s at stake? Considerable time and intellectual effort to develop a really good standard for A/V metadata, something everyone in the moving image community needs. Plus a certain (large) degree of credibility, because CPB was leading the PBCore effort and now we’re in some danger of abandoning PBCore. Like we somehow just forgot about it. With this project, Public Broadcasting has been a hero to the librarians and archivists, but it looks like we’re dropping the ball just when everyone wants to play.

So let’s some of us carry the PBCore torch for the next bit while pushing for further CPB action. We might have to get a bit militant. When someone ticks off the librarians, you don’t want to see what happens. Or maybe you do…

Jack Brighton
WILL Public Media

PubForge is a non-commercial group of volunteers working to develop a set of collaborative open-source tools to serve the needs of the public broadcasting community. Toward that end, we are surveying station staff, producers, developers and other interested parties to determine what the current state of the art is in public media, what tools and resources are most needed, and what collective capabilities are available to draw upon to meet these needs through collaborative effort. Please take a few minutes from your day to complete this survey.

A digest of the results of the survey, minus personal identification and contact info of participants, will be available in September at PubForge.

Bill Haenel, Dale Hobson and Jack Brighton at Public Media 2008 (Photo Credit: John Tynan)

I’ve worked as a webmaster in public broadcasting for almost a decade. And over the last several years, I’ve seen a slow, pragmatic shift towards increased collaboration in online ventures between local public broadcasting stations and national organizations and producers as evidenced (in NPR’s Podcasting initiative, their relaunch of NPR Music and) in the ongoing Election Collaboration. At the recent Public Media Conference in Los Angeles, Bruce Theriault recalled how he motivated national organizations collaborate around the 2008 Election by saying, “we will only fund this project if there is collaboration across silos - and if its shared with stations.”

And, while this initiative has exercised great strides towards increased cooperation across numerous organizations, it is my opinion that we still have yet to come into our own as a network. As Bruce Theriault says again “we need to get out of the walled garden of public media and allow the public and other institutions a chance to play.” To a greater or lesser degree, these initiatives are still fairly centrally controlled and (aside from the NPR podcasting initiative and Public Media Metrics Project) have yet to truly leverage the unique characteristic of public broadcasting as a distributed, network in general, and more specifically the potential of an open source model of collaboration.

Imagine what we could accomplish if we leveraged the combined efforts of the fifty or so interested and capable web professionals all working at public broadcasting stations (not to mention the larger community of programmers and the general public, many of whom happen to love public media… a lot) who would welcome the opportunity to work together towards a number of shared solutions (many of which would have clear benefits to our audience directly).

With that in mind, two weeks ago, I sent out an email to a half-dozen of my colleagues citing my reasons for why it would be useful to begin collaborating around open standards, common practices, and a common software and scripting platform distributed through an open source license. My email went something like this. I proposed that we form:

1) A co-op for public broadcasters to share code - and costs - where we agree on a similar solar system of scripting resources and practices - where we leverage upon an existing codebase and (ideally) share our efforts among stations and among the open source community as well. When needed, we can collectively raise money to pay outside developers to tailor code to our
needs and - where we are literally invested in the success of this venture and of each others sites.

2) Rather than relying on our own expertise alone to steer this ship, I propose we talk with a hosting provider or a organization like NPower or NTen or grassroots.org (which specializes in supporting non-profits with their technology needs) about providing hosting and (some of) the ongoing support. This way, we could focus on initiatives which we could band together and leverage shared code and programming costs and not have to be reliant on each other for the maintenance of the system.

Anyone who has gotten to know me over the years knows that this is my baileywick (As evidenced from This post from last year’s conference. However it turns out now this idea is not just important to me… or to a few of my friends… just recently…

The Knight Foundation awarded a $327,000.00 grant to Quiddities to develop an open source website and content management tool for KUSP as a model for public radio stations nationwide.

I’m sure the bright folks at the Knight Foundation and KUSP had given this idea a great deal of thought… and I know there are a ton of other excellent ideas percolating within public broadcasting right now as well… but I can’t help feeling like the guy who happened to step in front of the right parade at the right time. What I’m trying to say is this, I can’t take any credit for this grant, but I can say that I’ve seen it coming, and I could not be more delighted for us all!

With that in mind, as a first step in enlisting input from other stations on this project, Steve Laufer from KUSP got on the phone with Bill Haenel from the Integrated Media Association, Dale Hobson from North Country Public Radio, Jack Brighton from WILL, John Tynan (me) from KJZZ, and Matthew Tift from Wisconsin Public Radio to begin to discuss how we might work together on such a project and what first steps we would begin to take.

Some of the tasks that came out of today’s call were to:

• Set up a wiki to generate and focus some specific questions about what people would want to see in an open source CMS for their radio or television station.
• Create a survey to identify and prioritize features of the proposed CMS.
• Identify the skills and interests of people wanting to be involved in this project.
• Identify what existing project people would be willing to contribute to this endeavor.
• Identify how this could promote participation (and interoperability) between stations and national producers and our audience.

Please know that these initial impressions of the project are more personal than they are official. Aside from our conference call, I had only talked with Steve Laufer a few times between sessions at Public Media 2008. I have not been privy to the discussions between KUSP, Quiddities and the Knight Foundation. However, I know I’ve been thinking about this for a long time. I am sure that there is more than a handful of people (like me) to whom the principal parties can turn to for assistance and who will be be happy to devote their energies to the project’s success.

Cross posted on my personal site at johntynan.com.

Today, in subscribing to the Podcast for PyCon 2008, I noticed this entry:

“It almost seems like a joke: a family-owned newspaper in Lawrence, KS (population 80,000) releases an open-source web framework. It’s not a joke, of course: today Django is an increasingly popular web development platform. As an open-source community Django has been incredibly successful; in Tim O’Reilly’s OSCON keynote, he called Django “the new face of open source.” But it’s often unclear how we got here. How did a couple of programmers at a newspaper convince management to contribute to the open-source ecosystem? How does the company justify the time its developers spend on open source? And how have we as individuals and as a business had to adapt to become better open source developers?”

I was then like, “Huh! A family-owned newspaper in Lawrence, Kansas? That sounds familiar?!! Could it be? Yep, it is… Rob Curley the 2005 IMA Conference Keynote Speaker who “blew the roof off the Parc 55 with a dynamic presentation, illustrating his strategy of “hyper-localism.” Curley is one of the most decorated newspaper web directors in the United States. Some called it the best keynote speech–ever…” You can read more about his keynote speech here.

I remember coming away from the conference saying “I want to do what he does!” What an exciting, energizing person, who’s making a difference in his community and in the media industry. And now to find out that he’s doing it using open source technologies, and releasing a cool new web application framework based on python to boot! I find myself saying again… “I want to do what he does!”

I know there was some talk at last year’s conference about using Pubforge.org to support open source projects both within public broadcasting as well as independent media producers from beyond broadcast.net. I know too that, in addition to Pubforge.org , there’s always the Public Broadcasting Open Source Best Practices google group. There’s also the successful open source project from WNYC and KCRW, the East West Audio server. And there’s been collaborations that have not necessarily been open-source, like the momentum around the IMA’s with the Public Media Metrics project. But I wonder if the public broadcasting community could better support open source projects?

Tell me, what do you think it will take to foster a vibrant open-source community within public broadcasting? Tell me, what do you think it would take to have some real momentum around open-source software projects?

For those of you who came away from Rob Curley’s 2005 IMA Conference Keynote Speech and felt, like I did, that “I want to do what he’s doing!” And for those of you who would like to do this, like Rob, using a collaborative, open source approach, tell me, is 2008 the year for us to get organized? Is this year’s Public Media conference the place for us to start?

Tags: opensourcebroadcasting, publicbroadcasting

Just got off the phone with Seth Gotlieb (formerly of optaros.com, now at contenthere.net ) he had presented at IMA2007 as part of the discussion on choosing a cms.

Seth had some great advice that helped me form my thinking about how I should proceed as a technologist as well as how the folks rallying together at pubforge.org might best proceed as a group.

As someone who has built a good part of a station site using a particular brand of open source technologies (let’s say, I’ve chosen to drive our station around in the open source equivalent of a Ford), I will be facing a decision, given that there seems to be some considerable intertia in the Chevy camp. But now may not be the time to jump from one moving car to another, at least not yet.

Seth suggested that some good first steps would be for us to:

• Identify group of stations (or individuals) who are willing to work together around a specific (technology) or goal.
• Arrange for a week-long training session for the group in a single physical location. Either decide which city you would like to hold this as a group, or decide the city based on where the training is being held. (For plone users, he suggested contacting Joel Burton about a Plone Bootcamp — for drupal users, he suggested talking with Jeff Robbins at lullabot.com).

He went on to say that the benefit of getting together in the same place would:

• be an indicator of commitment - those who would be willing to travel would be more invested
• Getting out of the office would allow us to focus better
• It would be an opportunity to forge bonds socially and increase networking opportunities

He suggested we identify which projects are currently in development (such as the drupal stations modules project, or find/start a broadcasting equivalent to the ploneforartists project). He suggested we identify which aspects of these projects we would like to see improved or added upon. He suggested that we could add an economy of scale by either collaborating on code as a group, or by pooling our cash to pay for additions to the codebase.

He suggested that we check into the pricepoints for training. If we have x number of participants, what will it cost us?

He suggested, in looking for people who would be willing to attend the training, that we should start with the folks who initially put the module together, for instance the drupal station modules were originally designed for KPSU, a college radio station in Portland, Oregon. Maybe this station would be a good place to start with a partnership, and then look outward from there.

I guess that leads to the question, is there a listing of folks from the latest IMA conference who were interested in using Drupal, Plone or alfresco (or perhaps frameworks such as jboss or ruby, or django — or even closed source cms’ like Jack Brighton’s work with expression engine) the list goes on? Do you think such a list should be put together at pubforge.org?

To get a better idea how these discussions might be beneficial to Seth in his work, I asked “what was in it for him?” He replied that he wanted to keep tabs on the progress of these initiatives, that he would be interested in helping us form an organization, for helping us decide how such an entity would be structured, and how we are going to go about making decisions. His emphasis is in identifying the requirements for a product, in product selection, in enabling developers to work together and enabling companies work together using collaborative techniques / open source tools. Perhaps we’ll draw on his expertise again further down the road?

Tags: beyondbroadcast, ima2007, opensourcebroadcasting, pubforge

P.S. I did not realize that the blog at pubforge.org was setup, so I had posted this at the old site at webresources.org. Here are some comments on this post that we’ll want to move over here as well:

Jeff Robbins Says:

I heard my name mentioned and I figured I’d say “hello!” Yes, if you’ve got any questions or need help, we’d be happy to point you in the right direction and/or help you out directly. I’ve got a lot of interest in audio and broadcasting and we use many of the modules that Andrew Morton has written for KPSU on Lullabot.com.

Tim Olson Says:
Perhaps there is an upcoming developer conference/training that is focused on one of those (Ruby on Rails, Plone..) that we could tie this to?

We’re in the process of reconstituting the blog so it includes a broad group of public media practictioners. As a way of getting things started, Keith Hopper put together some language summarizing potential goes for the blog.

Hi everyone, Keith Hopper here (ok, how do we turn on bylines?). I’m a stickler for objectives. A group blog felt like a great idea, but I wanted to better understand what we might hope to accomplish. Here’s what we’ve come up with so far. Please suggest additions.

• Encourage public broadcasters to experiment and implement participatory media ideas learned from examples shared within the industry. Sharing examples will also help fledgling public media experiments by driving traffic to concepts that need attention, advice, and corrective action
• Stimulate original thinking by sharing trends and examples from outside the public broadcasting industry that may be relevant
• Provide a spotlight for individuals to introduce new, innovative ideas and get a dialog going about what public media might look like moving forward
• Help build a language and community around forward-thinking public media – make important connections across organizations to stimulate real work
• Demonstrate that public media professionals can be ahead of the curve – let’s lead the thinking not react to it
• … and other goals that I’m sure we can all figure out

-Keith Hopper

Feedback is most welcome.