This projects aims to share text content from online participation processes by sharing crawlers instead of the text contents themselves which might not be possible due to legal reasons. The text contents of the individual online participation plattforms will be downloaded via Scrapy into a JSON file.
This will install all required dependencies:
make buildmake run
scrapy crawl <dataset>| Dataset name | Link | # Suggestions | # Comments | # Total | Crawl time | Command |
|---|---|---|---|---|---|---|
| Bonn 2015/2016 | bonn-macht-mit 2015/2016 | 335 | 2937 | 3271 | 27 seconds | scrapy crawl bonn2015 |
| Bonn 2017/2018 | bonn-macht-mit 2017/2018 | 55 | 109 | 164 | 5 seconds | scrapy crawl bonn2017 |
| Bonn 2019/2020 | bonn-macht-mit 2019/2020 | 232 | 600 | 832 | 1 minute | scrapy crawl bonn2019 |
| Bürgerbudget Wuppertal | buergerbudget.wuppertal.de | 261 | 4 minutes | scrapy crawl wuppertal2017 | ||
| Köln 2012 | buergerhaushalt.stadt-koeln.de/2012/diskussion | 594 | 1879 | 2473 | 18 minutes | scrapy crawl koeln2012 |
| Köln 2013 | buergerhaushalt.stadt-koeln.de/2013/buergervorschlaege | 592 | 3095 | 3687 | 5 minutes | scrapy crawl koeln2013 |
| Köln 2015 | buergerhaushalt.stadt-koeln.de/2015/buergervorschlaege | 631 | 1855 | 2486 | 10 minutes | scrapy crawl koeln2015 |
| Köln 2016 | buergerhaushalt.stadt-koeln.de/2016/buergervorschlaege | 827 | 1314 | 2141 | 9 minutes | scrapy crawl koeln2016 |
| Leitbild Bad Godesberg | bonn-macht-mit.de/dialoge/onlinebeteiligung-zum-leitbild-bad-godesberg | 556 | 698 | 1254 | 2 minutes | scrapy crawl badgodesberg |
| Mängelmelder Braunschweig | mitreden.braunschweig.de/node/1358 | ≥ 3220 | 34 minutes | scrapy crawl maengelmelder-braunschweig | ||
| Nahverkehrsplan Ulm | zukunftsstadt-ulm.de/dialog | 498 | 619 | 1117 | 1 minute | scrapy crawl nahverkehrsplan-ulm |
| Raddialog Bonn | raddialog.bonn.de/dialoge | 2331 | 2425 | 4756 | 16 minutes | scrapy crawl raddialog-bonn |
| Raddialog Koeln | raddialog-ehrenfeld.koeln.de/dialoge | 378 | 277 | 655 | 2 minutes | scrapy crawl raddialog-koeln |
| Raddialog Moers | raddialog.moers.de/dialoge | 463 | 300 | 763 | 3 minutes | scrapy crawl raddialog-moers |