Python3 - interacting with HTML pages help

+4 James Kon · October 30, 2015
I need a plugin that allows me to do these things:

  • Read HTML codes

  • Simulate the clicking of a button

  • Be able to submit forms from a HTML page in the background of the computer

  • Can enter text into input boxes

  • Able to simulate the pressing of buttons, like pressing the enter button



Making it be in the background, and not maximized is optional, but would prefer that the user not see it.inter

Post a Reply

Replies

- page 1
Oldest  Newest  Rating
0 alex constantin · October 30, 2015
I have never seen a plugin that can do this but I know a way of posting data on a form that uses POST method: 
You need the requests module, which you can download using pip install requests



import requests
formData = {'username': "myusername", \
   'password': "mypass", \
   'otherForm': "whatever"
}
url="https://urltoformpage.com"

headers = {"add headers":"here"}
r = requests.post(url, data=formData, headers=headers)
0 Halcyon Abraham Ramirez · October 30, 2015
@alex constantin

you can post data to forms with request
but how do you simulate the button press with it?

and didn't I mention selenium before ? :/
0 sfolje 0 · October 30, 2015
Here things get dirty.
I think (, im not sure, but) technically you dont really need to click the button (or input text etc.), to access a website behind the form, you just need to perform post method (or some kind of command, that html+javascript uses in background) to process webpage form.
Some times there are some kind of sessions, or cookies, so you cannot access the needed webpage without a session id or key  etc. so you need to first get cookie (or session) id (in my experience manually by clikcing the button) and put it in headers as @alex constantin mensioned. Things can get tricky pretty easily.

tool: requests
0 James Kon · October 30, 2015
Halcyon Abraham Ramirez,

Yes you did. However, selenium had a lot of trouble reading the page coding. At least for me. I tried multiple times to get it to read the page title. It also didn't seem to have many good tutorials on using it.
+2 alex constantin · October 30, 2015
A tutorial explaining my previous post:  https://www.youtube.com/watch?v=eRSJSKG4mDA
0 James Kon · October 31, 2015
alex constantin,

In the tutorial it shows you looking at the token after you had logged in, the website I am using does not have a login/ name passed through, and if it does it is hidden. Is there a way to search all the information passed through? As there is quite a bit to look through.

Edit: I found the problem, directly after logging in, it would take me to another page before it took me to the actual website. I got it though.
0 James Kon · October 31, 2015
I went back to selenium, as well as watched the tutorial Alex gave me. I ran into another problem.

After getting the source code from the website, to check the title of it I am doing this:


page = c.get(url)
page1 = page.content
var1 = 0
while "Just a moment..." in page1:
var1 += 1


Page1 is the variable with the source code of the website by the way, however when I run it I get this error:

while "Just a moment..." in page1:

TypeError: a bytes-like object is required, not 'str'


I tried looking on other websites for help of this, but in their examples of turning things into a byte object doesn't involve variables.
0 alex constantin · October 31, 2015
Try converting the page1 variable to a string maybe? Not sure, but try str(page1)
0 James Kon · November 1, 2015
str is refering to it being a string, it already is a string. It needs to be converted in to a bytes-like object, whatever that is...

I did try that by the way, I got the same error:

a bytes-like object is required, not 'str'
+2 alex constantin · November 1, 2015
Try this:

phrase = bytes("Just a moment...")

while phrase in page1:
#do code
  • 1
  • 2

Python

107,046 followers
About

This section is all about snakes! Just kidding.

Links
Moderators
Bucky Roberts Administrator