Sitemap is a CKAN extension which will provide sitemap.xml which will inform search engines about available URLs. Note that this extension doesn't provide any modifications to robots.txt provided by CKAN so without any modifications the URL exclusion provided by robots.txt is in use.

The extension has been tested with CKAN 2.4.1

Activate your CKAN virtualenv:

source /usr/lib/ckan/default/bin/activate

Install ckanext-sitemap:

pip install -e git://github.com/kata-csc/ckanext-sitemap.git#egg=ckanext-sitemap

Install its dependencies (you will also need the libxml2 and libxslt C libraries):

pip install -r /usr/lib/ckan/default/src/ckanext-sitemap/requirements.txt

Add ckanext-sitemap to your CKAN configuration file (e.g. /etc/ckan/default/production.ini):

ckan.plugins = ... sitemap

Finally, restart CKAN:

sudo service apache2 restart

Once installation is complete and you have restarted CKAN the sitemap will be available at /sitemap.xml.