I'm fascinated with how people get their information each day. While social media is playing an increasingly significant role, newspapers are still how many folks get their information. I'm interested in understanding the role newspapers still play (or don't) in crafting the narrative of the country, and I wanted to put my finger on the pulse of the narrative heartbeat comming out of papers across the US.

I began this journey recently by looking through the front pages at the Newseum. I worked to automate the extraction of headlines from each of the papers, trying to craft a snapshot of the narrative for our country each day. The PDFs proved difficult to reliably get the top news stories of the day, as the words aren't always horizontally or vertically aligned in a algorithmically sensible way.

After going down this rabbit hole for a few weeks I decided to pull a list of the domains for 2000+ newspapers from Wikipedia. Which I then harvested and parsed an RSS feed for each publication wheneveer possible (not all have RSS). Out of that I was able to get roughly 900 RSS feeds, which produce about 400-500 reliable stories for each day. From this list of stories I randomly grab a single headline for inclusion as the narrative for each day. It isn't always the top story, but it is a story from that particular publication, city, and state.

I have automated the discovery of feeds, the pulling of their contents each day, and the random selection of a single story for inclusion in what I'm calling the US Narrative for Today. I've also managed to pull a random set of images from my algorotoscope work, adding a (random) visual dimension to each story published--the challenge now is to grow the number of images to limit the repetition of images across the stories. I'm not sure what is next, but I will keep harvesting, publishing, and seeing what else I can derive, automate, publish, and share from across the US news landscape. I will keep polishing how it assembles stories. Ultimately I'm not in control of the narrative for this story, I'm just following where the algorithmic thread takes me.

- Kin Lane