-
Notifications
You must be signed in to change notification settings - Fork 0
DeCarabas/sociallists
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
# Setting up the database - Install postgres - Create a database named 'sociallists' - In python, do: >>> from sociallists import db >>> db.Base.metadata.create_all() # Some notes on asynchrony I spent some time trying to convert this codebase to asyncio so that feed updates can be SUPER PARALLEL. What I discovered is that if I do this the HTTP stack begins to misbehave and timeout and stuff, and I'm not entirely sure why. So aiohttp doesn't seem to be a good choice for me. Using coroutines but a fundamentally blocking HTTP stack (asyncio + requests) doesn't actually buy me anything because all my threads will be blocked on the same fixed number of HTTP requests anyway, so it will be a lot of complexity for absolutely no efficiency gain. (In addition, the async/await stuff is currently very ugly and the docs are complicated and I just like this one better for now. So we're sticking with the slow but useful. Rewriting the feed update process to be massively parallel is still something I want to do but it will have to wait.) # TODO: UI: - Error boxes - Enable jumping between refresh sessions: - Mark the top refresh session on all rivers when we start a refresh. - Put navigation buttons in the boundaries between refreshes that jump up and down. (i.e., build a meta-FeedUpdate construct that is "refresh session"; it will be river-specific. This might be complex.) - Add/remove rivers - Add/remove feeds from rivers - Add/remove river sets - Re-order rivers Shipping: - Re-visit use of requirements and packages and vendoring and the like - Re-pack font awesome fonts - Use packed/minified/precompiled source if it exists Feed Stuff: - Extract enclosures and the like, store locally (in the DB) and present them in the river. (Then do UI work to expose audio enclosures and the like.) - Put river items in the DB so that we can efficiently re-run the processing steps for items that are out of the feed - For the future, we'll probably need options on individual feeds for processing to make them better. It is possible to over-engineer this so be careful. - Implement ATOM cache timeout and RSS skip hours and the like to poll more intelligently. - Implement pubsubhubbub - Clip titles that are too long. (Because seriously.) Bugs: - Can't add feed to river: stderr: 2016-07-18 10:15:23,113 127.0.0.1 - - [18/Jul/2016 10:15:23] "POST /wat/9 HTTP/1.1" 404 - - Fix Syblmoon bug where no title means broken rendering - b0rk's feed refreshes completely every time there's a new article; fix it! - Bug here: stderr: 2016-07-18 10:13:01,881 Error updating feed http://feeds.feedburner.com/buckblog: Traceback (most recent call last): File "/Users/doty/src/sociallists/sociallists/feed.py", line 244, in do_update_feed update = self.get_feed_update(history) File "/Users/doty/src/sociallists/sociallists/feed.py", line 191, in get_feed_update f, self.feed.next_item_id, update_time, self.http_session File "/Users/doty/src/sociallists/sociallists/river.py", line 186, in feed_to_river_update for e,i in zip(feed.entries, count(start_id)) File "/Users/doty/src/sociallists/sociallists/river.py", line 186, in <listcomp> for e,i in zip(feed.entries, count(start_id)) File "/Users/doty/src/sociallists/sociallists/river.py", line 152, in entry_to_river image = get_entry_thumbnail_image(entry, session) File "/Users/doty/src/sociallists/sociallists/river.py", line 131, in get_entry_thumbnail_image thumbnail_image = media.get_url_image(link, size, session) File "/Users/doty/src/sociallists/sociallists/media.py", line 37, in get_url_image thumbnail_url, http_session, referer=url) File "/Users/doty/src/sociallists/sociallists/media.py", line 74, in _fetch_url result = (response.url, response.headers['Content-Type'], response.content) File "/Users/doty/src/sociallists/venv/lib/python3.5/site-packages/requests/structures.py", line 54, in __getitem__ return self._store[key.lower()][1] KeyError: 'content-type'
About
Another run at the RSS fence.
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published