Skip to content

Converting the Ferguson Grand Jury transcripts into something more readable.

Notifications You must be signed in to change notification settings

tmaybe/ferguson-grand-jury-transcripts

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

41 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ferguson-grand-jury-transcripts

We have converted the 5000 page Ferguson Grand Jury transcript PDF into something more readable. This conversion makes the transcript searchable and linkable. We hope these features add to the national discussion about institutional racism and police brutality. We're using the SayIt service by MySociety.

Conversion

The Ferguson Grand Just testimony PDF is 5000 pages long. It was transcribed over 100 days by different people, so the formatting is all over the place. There is also a lot of redacted content. We've done the best we can to get these transcripts up and live. Done is better than perfect. Compare anything that looks weird to the official released PDF.

How You Can Help

  • Share with your local media
  • Find and share the most important parts of the transcripts.
  • Review the transcripts and write down any issues.
  • Improve our converter by writing code to fixing issues

Tasks

  • OCR the transcripts
  • Use pdf2txt.py to convert PDF to XML
  • Use parse_transcript_xml.py to convert XML to formatted txt
  • Use converted_text_to_akoma_ntoso.py script to convert the new formatted text to Akoma Ntoso
  • Upload to SayIt
  • Promote to media and activists

Raw Transcripts

http://graphics8.nytimes.com/newsgraphics/2014/11/24/ferguson-assets/grand-jury-testimony.pdf

OCR Transcripts

https://www.dropbox.com/s/67unqhdrb8jhgr0/Ferguson%20Grand%20Jury%20Testimony.pdf?dl=0

PDF to text script

parse_transcript_xml.py is a script that parses the XML files created by running pdf2txt.py (PDFMiner) on the OCR Transcripts.

The XML files generated by PDFMiner preserve formatting information which we can use to more accurately identify text attributed to individuals whose names have been redacted from the transcript, by examining text indentations.

The XML file of the full transcript can be generated with the command: pdf2txt.py -o files/ferguson_grand_jury_testimony.xml Ferguson\ Grand\ Jury\ Testimony.pdf

Final Transcripts

http://ferguson.sayit.mysociety.org/

Twitter Discussion

https://twitter.com/steiny/status/537297171255943168

About

Converting the Ferguson Grand Jury transcripts into something more readable.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%