We migrated Canary to a new server and have also placed it behind Cloudflare. Having done so, we have come across a problem where users who use the API may run into problems if they’re using it under Python.
Those of you who use Python 3 should be spared, but the official Canary API Python library was developed with Python 2.7 in mind. If you are running into problems where you’re getting error 500, you’ll want to ensure that all of your Python libraries are up to date and consider reading this StackOverflow question–you’ll want to make sure that none of this will break your existing configuration, et cetera.
As you might have noticed, Canary has had some hiccups in the past few weeks. This has been the result of some hardware problems that lead to the site being knocked offline.
To summarize, we moved Canary from one server to another (it’s running atop of KVM). The new server had a hardware RAID controller as opposed to a software one. For a few weeks, we had no issues until it suddenly gave out, leading to the server believing that the disks had outright disappeared. When this happened, the server was brought back online and a fix was being worked on to prevent this from occurring again–this was the result of a firmware issue within the RAID controller.
Unfortunately, it happened again.
What we’ve done instead is moved Canary back to its original hardware configuration and then had to restore the database as it had become corrupted in the process. It appears that everything is back to normal as of October 21st.
We’re monitoring for now and will update you on any changes. There may be an interruption this upcoming weekend but we will post on the front page if this should happen.
Canary has been going through a massive code rewrite the past few months and we’re getting closer to completing the work and testing things out. We figured it was time to let you all know about what is coming down the pipes for its update.
Everything listed here has not been implemented into the website as of yet but the next paragraph should indicate when things will start moving.
A new name
It should be no surprise that there are several other products in the information/wizard security space that use the name “Canary”. One of the consequences of this is that we’ve received support e-mails for products that we have nothing to do with. Because we are looking to at least be identifiable amongst others, we’ve gone ahead and adopted a new name.
We’re keeping silent on the new name for now but the new name will indicate that the overall site upgrade has occurred.
Search is what lives and breathes Canary. As such, one of the challenges we’ve faced is making it more effective so the data that you’re looking for is the data that you want to find. It has been no secret to us that the data that has been appearing within the search results has been erroneous and as a result we’ve wanted to completely diminish that.
The following changes will come into effect upon the new update (again, none of this is effect as of yet):
All search bangs will remain in effect but a new one will be added: “!tld”. This new bang will allow for searching of ICANN-approved TLDs but also for second-level domains such as “gc.ca”, “govt.nz” or “co.uk”. This will allow easier search for government servers as we’ve seen some use of the “!host” trigger for this purpose.
Generic searches will search extracted objects instead of keywords–meaning that keyword searches for the time-being will be retired. For the most part, searching for “220.127.116.11” will come up with results for any samples with the IP address contained within. An added feature is that searches for e-mail addresses will search for the hostname in addition to the e-mail address–so a search for “firstname.lastname@example.org” will show results for objects containing either “email@example.com” or “host.com”. Using explicit bangs will perform explicit searches–“!email firstname.lastname@example.org” will only return results for the e-mail address provided.
Still on generic searches, we are going to be returning keyword searches after this update but they will only be available to registered users. The use of the generic search will be relegated to its own bang. Keep in mind, registration is free.
Sub-searching is a new feature that is coming and will exist within the API and web-version of the website.
Searches will be limited to 10 results for unregistered users, 50 results for registered users, and 100 results for for the API.
We’ve also gone and removed data that is useless, meaning that anything that doesn’t contain extracted data has been put into a discard pile for us to review later. During this experiment we have identified new objects and are now extracting them. One of the objects we’re now extracting is Bitcoin addresses.
Additionally, some of the objects extracted will result in further data being included in the sample than what is explicitly there. This will allow for more useful searches should you be using a bang that wouldn’t otherwise find that information. We’ll reveal more details on this at launch.
At launch of the new Canary code-base, we’re keeping alerting as the same. However, the following is coming down:
Keyword-based alerts will be added. These will not permit for anything boolean-like, but it will allow for you to search for words within data that is found.
Regular expression-based alerting will be added as an option for alerts.
More details on the limits and what the costs will be will be revealed as we get closer to announcing this.
User account management
Two features are coming to user account management: the ability to reset your password in case you have forgotten it and the ability to disable your account (accounts that are disabled or inactivated will be outright purged periodically).
One of the features originally planned and promised for Canary was alerts. Finally we have delivered that and a few new features to the Canary website.
In this update, we have introduced the new Alert system. At the moment it is set up to only alert you if there is an incoming sample that has your Canary account’s e-mail address contained within–more on the future of this will be explained later. This feature is included with all accounts both current and future–we can outright disable it for your account if requested. Alerts are triggered shortly after the data has been analysed and will be tripped regardless of who submitted the data.
To access your alerts, you’ll need to login as normal (or register a free account and then login) and immediately you’ll be presented with an account page that shows an “Alerts” tab.
By default you should expect the above screen but in the event that there is an alert, you’ll be able to see and acknowledge them as follows:
In addition, you’ll be able to enable the e-mailing of alerts (disabled by default) so if an alert comes in you’ll be able to know without having to log into the website–this can be done by enabling it via the “configuring alerts” tab.
Down the road we’re working on the following for the site:
Keyword-based alerting in addition to the e-mail address you’ve signed up with. All accounts will have a maximum of three keywords and requests for more will be a paid feature. It should be kept in mind that alerts are not based on search queries so the usual triggers (!http, !ip, et cetera) will not apply here.
Regular expression-based alerting will be available on a pay-basis. Details are still being worked on this but we’ve had requests for regex-based searching in the past and would like to act upon it in some way.
API access to alerts which will be useful for those of you who are looking to integrate this into Splunk, LogRhythm, ArcSight, and so forth.
Stay tuned for more on alerting.
Public Data Submission
One of the things that we are aiming for is to make Canary more engaged with the community. As such, we’ve gone ahead and introduced a much simpler way to send us data. If you look on the top-right of the page, you’ll see a page available to you provided you’re logged in with an account:
You’ll be able to see on this page a list of your previously submitted items (if any of course) but in addition to that you’ll also be presented with a submission form where you can submit data:
Any data posted via this submission form is only accessible to those with registered accounts. Data shared via this service is by default not exposed to the public unless either it has already been submitted before and was made public or we decided to make it public for whatever reason. Your account details are attached to the submission but those will not be visible to the public regardless if they’re signed in or not.
All registered users have access to this feature and we look forward to having everyone contribute whatever they can. This is meant for one-off posts that are 100 KB or less but if you’re looking to upload mass amounts of data or anything larger, please let us know as we have much better mechanisms for submitting data.
We’re going to evaluate how this feature works out so expect changes if we deem them necessary.
We’re working on changing the search mechanism as it’s producing results that are either erroneous, a bit too verbose, or are simply not what users should and want to expect. This is a known problem and we’re working to resolve this. Making use of the triggers will allow for better results.
If you’ve been following social media as of late, you might have seen remarks about a new set of data that is floating about stating that PlayStation Network (PSN) along with 2K Games and Windows Live were all breached and that a dump of passwords has been made available.
But is this really the case?
The password dump was added to Canary yesterday and immediately we saw some interesting results. Here are the links containing the dump as it is (for analysis purposes most large dumps are split up):
While this requires free registration, viewing the ‘related’ tab on these entries (specifically parts 3, 4, and 5) shows a number of similarities between this dump and dumps belonging to booter forums (“booter” being a term used to refer to DDoS attacking) as evident in these screenshots:
The sources in question can also be viewed via these links (there are more than just these but it gives you an idea):
This sort of thing has happened before where dumps from various databases were either re-branded as something else or were co-opted by another group in an attempt to boost reputation. A perfect example of this is an old breach on a site run by the FBI, which has been stated to have been done by several different groups since it was first dumped–every few weeks to a few months we see the dump rehashed.
So should you be concerned about this dump? Well it comes down to what level of risk you want to take, but it should always be kept in mind that breach data may not be what it seems. If you think you’re at risk, do what is appropriate.