URS

Summary

This release fixes an open issue.

PRAW v7.3.0 changed the Redditor object's subreddit attribute. This change breaks the Redditor scraper. It would be nice if all the tools worked as advertised.

Full Changelog

Added

• Source code
  • In Redditor.py:
    • Added a new method GetInteractions._get_user_subreddit() - extracts subreddit data from the UserSubreddit object into a dictionary.
• Tests
  • In test_Redditor.py:
    • Added TestGetUserSubredditMethod().test_get_user_subreddit() to test the new method.

Changed

• Source code
  • In Redditor.py:
    • GetInteractions._get_user_info() calls the new GetInteractions._get_user_subreddit() method to set the Redditor's subreddit data within the main Redditor information dictionary.
  • In Version.py:
    • Incremented version number.
• README
  • Incremented PRAW badge version number.

Summary

• Introduced a new utility, -t, which will display a visual tree of the current day's scrape directory by default. Optionally, include a different date to display that day's scrape directory.
• Move CI providers from Travis-CI to GitHub Actions.
  • Travis-CI is no longer free - there is now a free build cap.
• Minor code refactoring and issue resolution.

Full Changelog

Added

• User interface
  • Added a new utility:
    • -t/--tree - display the directory structure of the current date directory. Or optionally include a date to display that day's scrape directory.
• Source code
  • Added a new file Utilities.py to the urs/utils module.
    • Added a class DateTree which contains methods to find and build a visual tree for the target date's directory.
      • Added logging when this utility is run.
  • Added an additional Halo to the wordcloud generator.
• README
  • Added new "Utilities" section.
    • This section describes how to use the -t/--tree and --check utility flags.
  • Added new "Sponsors" section.
• Tests
  • Added test_Utilities.py under the test_utils module.

Changed

• Source code
  • Refactored the following methods within the analytics module:
    • GetPath.get_scrape_type()
    • GetPath.name_file()
    • FinalizeWordcloud().save_wordcloud()
      • Implemented pathlib's Path() method to get the path.
  • Upgraded all string formatting from old-school Python formatting (using the % operator) to the superior f-string.
  • Updated GitHub Actions workflow pytest.yml.
    • This workflow was previously disabled. The workflow has been upgraded to test URS on all platforms (ubuntu-latest, macOS-latest, and windows-latest) and to send test coverage to Codecov after testing completes on ubuntu-latest.
• README
  • Changed the Travis-CI badge to a GitHub Actions badge.
    • Updated badge link to route to the workflows page within the repository.
• Tests
  • Upgraded all string formatting from old-school Python formatting (using the % operator) to the superior f-string in the following modules:
    • test_utils/test_Export.py
    • test_praw_scrapers/test_live_scrapers/test_Livestream.py
  • Refactored two tests within test_Export.py:
    • TestExportWriteCSVAndWriteJSON().test_write_csv()
    • TestExportExportMethod().test_export_write_csv()
• Community documents
  • Updated PULL_REQUEST_TEMPLATE.md.
    • Removed Travis-CI configuration block.

Deprecated

• Source code
  • Removed .travis.yml - URS no longer uses Travis-CI as its CI provider.

Summary

• Introduced livestreaming tools:
  • Livestream comments or submissions submitted within Subreddits.
  • Livestream comments or submissions submitted by a Redditor.

Full Changelog

Added

• User interface
  • Added livestream scraper flags:
    • -lr - livestream a Subreddit
    • -lu - livestream a Redditor
    • Added livestream scrape control flags to limit stream exclusively to submissions (default is streaming comments):
      • --stream-submissions
  • Added a flag -v/--version to display the version number.
• Source code
  • Added a new sub-module live_scrapers within praw_scrapers for livestream functionality:
    • Livestream.py
    • utils/DisplayStream.py
    • utils/StreamGenerator.py
  • Added a new file Version.py to single-source the package version.
  • Added a gallery_data and media_metadata check in Comments.py, which includes the above fields if the submission contains a gallery.
• README
  • Added a new "Installation" section with updated installation procedures.
  • Added a new section "Livestreaming Subreddits and Redditors" with sub-sections containing details for each flag.
  • Updated the Table of Contents accordingly.
• Tests
  • Added additional unit tests for the live_scrapers module. These tests are located in tests/test_praw_scrapers/test_live_scrapers:
    • tests/test_praw_scrapers/test_live_scrapers/test_Livestream.py
    • tests/test_praw_scrapers/test_live_scrapers/test_utils/test_DisplayStream.py
    • tests/test_praw_scrapers/test_live_scrapers/test_utils/test_StreamGenerator.py
• Repository documents
  • Added a Table of Contents for The Forest.md

Changed

• User interface
  • Updated the usage menu to clarify which tools may use which optional flags.
• Source code
  • Reindexed the praw_scrapers module:
    • Moved the following files into the new static_scrapers sub-module:
      • Basic.py
      • Comments.py
      • Redditor.py
      • Subreddit.py
    • Updated absolute imports throughout the source code.
  • Moved confirm_options(), previously located in Subreddit.py to Global.py.
  • Moved PrepRedditor.prep_redditor() algorithm to its own class method PrepMutts.prep_mutts().
    • Added additional error handling to the algorithm to fix the KeyError exception mentioned in the Issue Fix or Enhancement Request section.
  • Removed Colorama's init() method from many modules - it only needs to be called once and is now located in Urs.py.
  • Updated requirements.txt.
• README
  • The "Exporting" section is now one large section and is now located on top of the "URS Overview" section.
• Tests
  • Updated absolute imports for existing PRAW scrapers.
  • Removed a few tests for DirInit.py since the make_directory() and make_type_directory() methods have been deprecated.

Deprecated

• Source code
  • Removed many methods defined in the InitializeDirectory class in DirInit.py:
    • LogMissingDir.log()
    • create()
    • make_directory()
    • make_type_directory()
    • make_analytics_directory()
      • Replaced these methods with a more versatile create_dirs() method.

Release date: March 28, 2021

Summary

• Structured comments export has been upgraded to include comments of all levels.
  • Structured comments are now the default export format. Exporting to raw format requires including the --raw flag.
• Tons of metadata has been added to all scrapers. See the Full Changelog section for a full list of attributes that have been added.
• Credentials.py has been deprecated in favor of .env to avoid hard-coding API credentials.
• Added more terminal eye candy - Halo has been implemented to spice up the output.

Full Changelog

Added

• User interface
  • Added Halo to spice up the output while maintaining minimalism.
• Source code
  • Created a comment Forest and accompanying CommentNode.
    • The Forest contains methods for inserting CommentNodes, including a depth-first search algorithm to do so.
  • Subreddit.py has been refactored and submission metadata has been added to scrape files:
    • "author"
    • "created_utc"
    • "distinguished"
    • "edited"
    • "id"
    • "is_original_content"
    • "is_self"
    • "link_flair_text"
    • "locked"
    • "name"
    • "num_comments"
    • "nsfw"
    • "permalink"
    • "score"
    • "selftext"
    • "spoiler"
    • "stickied"
    • "title"
    • "upvote_ratio"
    • "url"
  • Comments.py has been refactored and submission comments now include the following metadata:
    • "author"
    • "body"
    • "body_html"
    • "created_utc"
    • "distinguished"
    • "edited"
    • "id"
    • "is_submitter"
    • "link_id"
    • "parent_id"
    • "score"
    • "stickied"
  • Major refactor for Redditor.py on top of adding additional metadata.
    • Additional Redditor information has been added to scrape files:
      • "has_verified_email"
      • "icon_img"
      • "subreddit"
      • "trophies"
    • Additional Redditor comment, submission, and multireddit metadata has been added to scrape files:
      • subreddit objects are nested within comment and submission objects and contain the following metadata:
        • "can_assign_link_flair"
        • "can_assign_user_flair"
        • "created_utc"
        • "description"
        • "description_html"
        • "display_name"
        • "id"
        • "name"
        • "nsfw"
        • "public_description"
        • "spoilers_enabled"
        • "subscribers"
        • "user_is_banned"
        • "user_is_moderator"
        • "user_is_subscriber"
      • comment objects will contain the following metadata:
        • "type"
        • "body"
        • "body_html"
        • "created_utc"
        • "distinguished"
        • "edited"
        • "id"
        • "is_submitter"
        • "link_id"
        • "parent_id"
        • "score"
        • "stickied"
        • "submission" - contains additional metadata
        • "subreddit_id"
      • submission objects will contain the following metadata:
        • "type"
        • "author"
        • "created_utc"
        • "distinguished"
        • "edited"
        • "id"
        • "is_original_content"
        • "is_self"
        • "link_flair_text"
        • "locked"
        • "name"
        • "num_comments"
        • "nsfw"