I'm trying to write a program that utilizes the Wikipedia API. As far as I can tell, the simplest way to use the API is to access an HTTP page with the requested command, for instance, this finds all links on the "Apple" wikipedia article. I want to implement commands like these into my Java program so I created the following snippet to fetch the data from a HTTP page:
URLConnection connection = null; // Connection to the URL data
InputStreamReader iSR = null; // Stream of the URL data
BufferedReader bR = null; // Reader of URL data
URL url = null; // URL based on the specified link
// Open the connection to the URL web page
url = new URL(link);
connection = url.openConnection();
// Initialize the Readers
iSR = new InputStreamReader(connection.getInputStream());
bR = new BufferedReader(iSR);
// Fetch all of the lines from the buffered reader and join them all
// together into a single string.
return bR.lines().collect(Collectors.joining("\n"));
This works fine for fetching data, however, it is very slow. Each fetch takes around half a second, which is unacceptable for my program, especially since processing the whole downloaded file only takes around 1/1000th of a second. Is there any way that I can somehow quickly download these small files?
The fastest way, if you don't mind not having the most up-to-date information (of course you could devise a way to keep the data up to date), would be to get a dump of the data.
This would allow you to create your own server that could return pre-formatted data, as well as returning multiple data items with a single request, making it a lot faster than parsing HTML from multiple requests.