Fix logging for non whitelisted IP

[2.1.0] - 2023-04-01

Fixed

• Properly log error and stop execution for non whitelisted IP

Changed

• Updated syntax in DiscordAction

Update to 2.0.0

[2.0.0] - 2021-07-13

Fixed

• Better protocol check in urlanalyzer (a377aee)
• Use sys.exit() instead of exit() (2d6cb67)
• Add missing parameter 'unique_matches' for match function (c9a2e99)

Added

• Implemented ExactWordAnalyzer to match words exactly rather than partially (08ebdbc)
• Implemented BasicAnalyzer.unique(), which can be used to filter out duplicate matches from BasicAnalyzer.match()
• Ability to enforce IP version for the connection to pastepwn (3483566)

Changed

• BREAKING: Dropped support for Python < 3.6

Fix: Undetected error for invalid IPs when connecting via IPv6

You need to whitelist your IP when you want to use the scraping API. The previous release did not properly tell you, when your IP wasn't whitelisted and you were using IPv6. This is now fixed.

1.3.1 - 2020-06-20

Fixed

• The PastebinScraper could not recognize error messages with IPv6 addresses.

Correctly implement IrcAction and bugfixes

This release focusses mostly on fixing bugs, but it also contains a new analyzer implementation and a rewrite of the IrcAction, which did not work previously.

1.3.0 - 2020-03-03

Added

• Implemented base64analyzer, which matches if a found base64 string decodes to valid ascii (b535781)
• Implemented IrcAction - the previous implementation was not working (546b87f)

Changed

• SaveFileAction now got a parameter to set the file ending and a template (c3d75f7)

Fixed

• Analyzers now check if a passed action is a subclass of BasicAction, which prevents issues such as #175
• The DiscordAction now also uses the templating engine - it was forgotten in a previous update (#176)
• The SyslogAction now also uses the templating engine - it was forgotten in a previous update (54d3652)
• The SaveFileAction does now store each paste in a different file as it should be (#179)
• The IrcAction did not send the correct data. This was fixed and eventually the action was rewritten from scratch (see "Added")

Do something with matched results

In this release I implemented a few new features, which require some wiki pages for better explanation. Here's a general list of new features:

1.2.0 - 2020-02-05

Added

• Analyzers can now return a boolean or a list of matched results
• Actions now get passed a list of matched results by the analyzer
• New Analyzer: PasteTitleAnalyzer - Analyzer to match Paste titles via regex
• New Analyzer: IPv4AddressAnalyzer - Match IPv4 addresses via regex
• Subclasses of RegexAnalyzer now got a method def verify(results) that can be overwritten to filter matches so you only return valid results
• EmailPasswordPairAnalyzer has now an optional parameter min_amount to specify how many pairs must be found to actually match
• Base64Analyzer got an optional parameter min_len to specify how long a detected string must be at least to actually match
• Logical operators for analyzers - you can now connect multiple analyzers with logical operators to specify more precisely when a paste should match (aed2dbf)

Changed

• Analyzers can now return a boolean or a list of matched results
• Actions now get passed a list of matched results by the analyzer and can
• IBANAnalyzer will now filter out wrong IBANs and return a list of validated IBANs if the validate parameter is set to True

Fixed

• Using non-capturing groups in regex for various analyzers. This is done so that the analyzer can return a matched string and at the same time it fixed some issues with analyzers not matching properly

This changelog was created with the help of Keep a Changelog

Bunch of new analyzers, tests and actions

During hacktoberfest 2019 there has been a huge interest in this project and people implemented tests, actions, analyzers and other stuff. I think that this all is worth a new minor release.

1.1.0 - 2019-11-11

Added

• Implement TemplatingEngine for filling template strings with content (8481036)
• Add custom request headers in request.py (5043e0c)
• Add flags to RegexAnalyzer to handle e.g. case insensitive matching (ddd0dca)
• logger object now usable from within any analyzer (d21532e)
• Implement logical analyzers (and/or) (94fc691)
• Implement listify method to create lists from a given input (e935122)
• Implement support for onstart handlers (25b5313)
• Create docker-compose file (83014be)
• New Action: TwitterAction for posting tweets when a paste matched (2056c3c)
• New Action: DiscordAction (eafdc1c)
• New Action: MISPAction (8dabe5d)
• New Action: EmailAction (9cfba96)
• New Action: IrcAction (fc1d1ab)
• New Analyzer: PrivateKeyAnalyzer (a8746f1)
• New Analyzer: DatabaseDumpAnalyzer (0aa63ad)
• New Analyzer: DBConnAnalyzer (e940630)
• New Analyzer: PhoneNumberAnalyzer (9ff58b9)
• New Analyzer: OriginKeyAnalyzer (d0d715d)
• New Analyzer: SteamKeyAnalyzer (27273a6)
• New Analyzer: UplayKeyAnalyzer (38097ac)
• New Analyzer: EpicKeyAnalyzer (da122da)
• New Analyzer: BattleNetKeyAnalyzer (8927204)
• New Analyzer: MicrosoftKeyAnalyzer (8927204)
• New Analyzer: AWSAccessKeyAnalyzer (ebc6eab)
• New Analyzer: AWSSecretKeyAnalyzer (d07021a)
• New Analyzer: SlackWebhookAnalyzer (c40c364)
• New Analyzer: GoogleOAuthKeyAnalyzer (fbfb8bf)
• New Analyzer: FacebookAccessTokenAnalyzer (bb51e3e)
• New Analyzer: Base64Analyzer (8d50fbe)
• New Analyzer: AdobeKeyAnalyzer (4e52345)
• New Analyzer: EmailPasswordPairAnalyzer (f0af9cb)
• New Analyzer: HashAnalyzer (87080c2)
• New Analyzer: SlackTokenAnalyzer (d686169)
• New Analyzer: MailChimpApiKeyAnalyzer (2e5302d)
• New Analyzer: MegaLinkAnalyzer (c884cb6)
• New Analyzer: StripeApiKeyAnalyzer (f9bd202)
• New Analyzer: AzureSubscriptionKeyAnalyzer (b010cb5)
• New Analyzer: GoogleApiKeyAnalyzer (635a5e4)

Changed

• Add pastebinscraper by default (d00fc83)
• Remove unused custom_payload from DiscordAction (7b13d75)

Fixed

• SHA hash analyzer can now accept multiple length hashes (494d1af)
• Use empty string if paste.body is set to None in URL- and IBANAnalyzer (09f6763)
• Include some changes when creating a sqlite file (0eb3504)

Re-downloading pastes if they are not ready and ignoring deleted posts

Before this update it was not possible to re-download pastes which were not ready for scraping yet. Also deleted posts were stored in the database and analyzed. This is fixed now!

1.0.16 - 2019-09-08

Added

• Perform checks on pastebin responses to catch errors (01f865e)
• If pastes are not ready for downloading, requeue them (01f865e)

New features - error handler, graceful shutdown, and much more

I finally came to checking my dev branch. I had some stuff up the pipeline and finally merged it. Have fun!

1.0.15 - 2019-09-04

Added

• Ability to search for multiple words in single WordAnalyzer (d2a7e09)
• Ability to restart running scrapers after adding a new one (de99892)
• Ability to register error handlers (1fae47e)

Fixed

• Check if paste is None before analyzing it (2fd7b39, f4bfa46)
• Broken behaviour for WordAnalyzer blacklist (df2dd5b)
• Reduced sleep time in order to shut down pastepwn faster (55bb18d)
• Add check in GenericAnalyzer if parameter is callable (781d6d0)
• WordAnalyzer not matching in case sensitive mode (8762ddd)

FIX: Database connection was broken

In this release I fixed some database (sqlite) related issues.

1.0.14 - 2019-09-04

Added

• Parameter for setting up storing of all downloaded pastes (e04f476)

Fixed

• Broken path to requirements.txt in setup.py (cc7edf4)
• Missing column 'syntax' in sqlite table layout (3fb3821)
• Broken variable substitution for sqlite statement (cf49963)
• Allow writing to sqlite db from multiple threads (f47ec62)

FIX: Remove unfinished API features

In the previous release I had a few broken/unfinished files in the code and a requirement for sanic (which I wanted to build the API with). I removed the requirement for it. Now the python version required should not be >3.5 anymore.

1.0.13 - 2019-09-02

Added

• Pastepwn got a logo! (57e6665)
• Use travis tag when building pypi package (bda3c7e)

Fixed

• Broken paths in setup.py (42eca9b)