I support this app in my free-time and at my own expense. Please consider offering a donation in order to promote continued development. You can donate on Paypal.

This app allows you to download contents of web-pages and index them. You can get the app here on Splunkbase.

Website Data Extraction: setup an input that will extract data from a web-page and get it into Splunk Data Preview: click "Preview results" on the input configuration page to get a sample of the what the output would look like before you save the configuration Configuration

Once you install the app, it will ask you to set it up on the app configuration page. The setup only contains options related to configuring a proxy server. If no proxy server is used, you can just press save.

You will need to create an input to define the websites that you would like to extract information from. You can setup a new input using Splunk's manager at Settings » Data Inputs » Web-pages. The most difficult part of configuring the app is making the CSS selector that will capture the data you want. See W3schools for information on how to create CSS selectors.

Tip: creating CSS selectors in Chrome

1. Navigate to the webpage to be splunked
2. Right click on the page > select “Inspect element”.
3. You will then have a new section on the right of the web page, containing the elements of the page
4. Move your mouse down this text until the specific part of the page you are interested in highlights.
5. Right click and select “Copy CSS Path“
6. Go back to the Splunk app setup page and paste this into the “Selector” field” (Example: #TIRatesDisplay_tblTermDeposit)
7. Repeat this for other elements on the same page you wish to ingest separated by commas (Example: #TIRatesDisplay_tblTermDeposit, #TIRatesDisplay_tblLoan)

You can also use the new wizard form to make creating the selector easier:

You can usually ignore the "Output" section. This is only necessary if you want to name the fields that the input will get based on content within the page (see "Can I use attributes to set the field names?" for details).

The "Authentication" can be left blank unless the web-page requires authentication.

See the links below for answers to frequently asked questions:

Can I specify more than one selector (to match different things on a single page)?

Can I use attributes to set the field names?

This project is open source.

See GitHub for the source or LukeMurphey.net for more information.

The pacakge is available on Splunkbase