We aim to support a wide range of users to access legislation data and use legislation.gov.uk in different ways, including users using the site in a professional or personal capacity, institutional and commercial users, and crawlers (automated programs which systematically scan websites). We welcome anyone searching and browsing legislation, archiving the website or indexing it for search engines, and we encourage the downloading of data for re-use.
The non-human traffic we receive on legislation.gov.uk falls into several categories:
The legislation.gov.uk website contains over 300,000 documents and there are often many hundreds of ways users can view (sections of) each one. Some individual documents can also contain more than 10,000 pages worth of text. You can find information about our data and formats available through our API on our Formats page.
Some users can, accidentally or intentionally, place a greater demand on website services than others. For instance, users running automated scripts that request a very large number of pages within a short amount of time.
The aim of this policy is to ensure fair access to legislation.gov.uk for all users whilst providing a suitable level of performance. We apply this usage policy to all website users. We do not provide or offer priority or preferential treatment to any group of users or type of usage.
We want our website to be as open and inclusive as possible, but we also want it to be safe, secure, and available to everyone. For this reason, we have established a few ground rules for you to follow when using our website. Failure to comply with these rules may lead to denying your access to the website (see Restrictions for more information).
If you are not using a browser, you must identify yourself, your web scraper or crawler with a legitimate user agent string in the User-Agent header of your request. We do not accept anonymous user agents (see Restrictions), and so your user agent must contain a clear identifier.
We strongly recommend that you add contact details to your user agent—either an email address, or a URL for a page that explains what your bot does and how to contact you. This means that we can contact you if there is a problem with your bot. You can find more details on the information we collect about you online and your rights by reading our Privacy Notice.
The following is an example of a user agent containing a good identifier with an email address:
My-Bot (contact@yourdomain.net)
The following is an example of a user agent containing a good identifier with a link to a web page:
My-Bot (https://yourdomain.net/mybot.html)
You must follow the rules in our robots.txt file (http://www.legislation.gov.uk/robots.txt). The robots.txt may specify how frequently and which pages you can or cannot crawl. These rules may change over time as we review this policy (see Changes to this policy), so make sure to check them regularly.
You must not exceed our request rate limit of 3,000 requests in any 5 minute period for each IP address. If you exceed the rate limit, our API will block your IP address until the average number of requests over the previous 5 minutes falls below the rate limit.
Use a reasonable crawl rate to prevent overloading the website with requests. Follow the crawl-delay setting in our robots.txt, if provided. If the crawl-delay setting is not provided, use a conservative crawl rate (e.g. 10 requests per 5-10 seconds).
If you want to:
you should use a different way to download content.
If you believe that:
please contact us prior to taking any action.
In order to preserve the security, stability, availability or integrity of our website, we reserve the right to suspend or modify any user’s access to our website if the user:
Furthermore, note that:
We may revise this policy at any time, with or without notice, in order to maintain an acceptable level of functionality and performance of the website and API (e.g. in the event of significant changes to traffic).
You may want to check this page and the robots.txt page from time to time to take notice of any changes we make.
Some of the statements contained in this policy may also be superseded by statements published elsewhere on our site.
If you find it difficult to follow our fair use policy, there may be an alternative way to get the content you need:
We encourage users to use these services where appropriate to their needs, instead of crawling the entire website or API. Doing so may save you (and us) time and money and free up server resources for other users.
We are happy to assist data users to acquire our data in a responsible fashion.
If you need any further information about this policy, you feel you have been blocked in error or you are otherwise having difficulties, please contact us.