Is there a Java Web Bot Tutorial Anywhere?

+1 Nathan Lile · July 21, 2014
I posted this earlier... Im trying to learn how to interact with the web in Java and I haven't been too successful on my own. JSoup has simply confused me.

(I would really like to do automated searches and record results. For instance, auto search for some keywords and save URL matches to a txt file.)

Post a Reply


Oldest  Newest  Rating
-2 Brandon Elliott · July 21, 2014
Seems like PHP would be a better language for this
0 Nathan Lile · July 22, 2014
I can't really ask a specific question. I'm just confused by the overall documentation and I don't have any knowledge of web based java. All I've done is basic PC applications and data structures. I've never dealt with web at all.

I'm sure there are languages that are better than Java for web scraping, but I'd really like to learn in Java. However, if anyone knows of a really good tutorial in another language for something like this, I would be really appreciative!
+2 William Frost · July 22, 2014
I am a great fan of web scarping/spidering, I found something for Java. It worked for me. Gl :) 
0 Colin James · August 3, 2014
"I would really like to do automated searches and record results."

Very easy to do.
First, find an entry point for your query. Be it a GET request performed in the URL. So let's say we have a page such as:
which accepts 1 get variable called query, for example. 
So when we search for, I dunno, "cake". We see the results and the URL is:
Something like this would suffice:
// Initiating these objects can sometimes throw exceptions, so we will catch them all with this try catch.
try {
// Create a new URL object. Note: If your URL is going to contain spaces then you should use: URLEncoder.encode("urlblahblah", "UTF-8");
URL url = new URL("");
// Create a connection so we can fetch input data
URLConnection con = url.openConnection();
// Create a bufferedreader to read the input stream (what the URLConnection's HTTP request openConnection() responds with)
BufferedReader br = new BufferedReader(new InputStreamReader(con.getInputStream()));
// Create a holder for the current line to avoid missing the first line of the source
String curLine;
// We could just append to string but I don't like primitives
StringBuilder builder = new StringBuilder();
// Read each line of response until it's done (equal to null)
while((curLine = br.readLine()) != null) {
// Close up leaks
// Create string object equal to the whole source created in the while loop
String pageSource = builder.toString();
// Output to console
} catch (Exception e) {

That's a good starting point. From there you'll have to parse the response. You can get good web apis that respond with JSON or XML which makes it simple to parse. If it's not a data notation response (i.e. a normal page) then you can make use of Java's easy regex functions.

Good luck!
0 Sora Innosia · September 15, 2015
You can create Web Bot easily using a tool call ScrapperMin (available for Android and Windows)
download at

Search in Play Store : ScrapperMin
you will need basic learning curve of the syntax and basic knowledge of HTTP protocol. If you need to learn it, its easy, run a browser in PC and press F12, go to network tab, and click a link in browser and observe what browser send to the website in order to get the page, or in order to do Login, posting, ScrapperMin let you easily post all those data back to server without writing a bunch of code like above
+1 William Frost · September 15, 2015
@Sora, I don't think he wants to do nothing... don't spam here
0 Oceanblue digital · September 16, 2015
Yes you can check out this video -

This tutorial demonstrates how to get started in programming Java Bots.
0 Oceanblue digital · September 16, 2015
Yes you can check out this video -

This tutorial demonstrates how to get started in programming Java Bots.
  • 1

Java / Android Development


Very popular language used to create desktop applications, website applets, and Android apps.

Bucky Roberts Administrator