[selenium-users] Unable to archive webpages with Selenium and pythonwayback

[s***@gmail.com]

I've been redirected from this github
issue https://github.com/SeleniumHQ/selenium/issues/6716 to here.
I waited for up to two minutes, but the stale error doesn't go away. I've
update the bug report from the github one to reflect changes

## 🐛 Bug Report

I'm trying to click on a webpage, which is loaded via pywb
https://github.com/webrecorder/pywb so that all content get archived for
later.

If I'm loading the page directly without pywb, I am able to click on a
botton on the page from python.
Trying to do the same with webrecorder, just errors out.



To Reproduce

1. Create a wayback-server with pywb under these
instructions https://pywb.readthedocs.io/en/latest/manual/usage.html#getting-started
2. Open a webpage from selenium with the archive
3. Try to click on the "Load earlier"-button via xpath


Expected behavior
On the Webpage, the button gets clicked and more comments are loaded. (This
is the case when the minds.com-url is directly loaded)

Actual behavior
Script just errors out and the button hasn't been clicked
<pre>
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File
"/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webelement.py",
line 80, in click
self._execute(Command.CLICK_ELEMENT)
File
"/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webelement.py",
line 633, in _execute
return self._parent.execute(command, params)
File
"/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py",
line 321, in execute
self.error_handler.check_response(response)
File
"/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py",
line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale
element reference: element is not attached to the page document
(Session info: chrome=70.0.3538.67)
(Driver info: chromedriver=2.41.578700
(2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 4.18.0-2-amd64
x86_64)
</pre>

Test script or set of commands reproducing this issue
pywb installation
<pre>
$ pip install pywb
$ wb-manager init archive
$ wayback --record --live -a --auto-interval 30
</pre>

### script connecting to pywb
<pre>
from selenium import webdriver
from time import sleep
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("http://localhost:8080/archive/record/https://www.minds.com/newsfeed/800266828058890240")
(...)
while driver.find_element("xpath","//minds-comments/div/div"):
print("click")
driver.find_element("xpath","//minds-comments/div/div").click()
print("sleep 3 sec")
sleep(3)
(...)
</pre>

Environment

OS: Debian 4.18.10-2
Browser: Chromium
Browser Driver version: chromedriver=2.41.578700
Language Bindings version: Python3 selenium==3.141.0

[s***@gmail.com]

Update, on IRC it's been told that it's an iframe, but I can't switch to
the iframew for some reason

I've been redirected from this github issue
https://github.com/SeleniumHQ/selenium/issues/6716 to here.
I waited for up to two minutes, but the stale error doesn't go away. I've
update the bug report from the github one to reflect changes
## 🐛 Bug Report
I'm trying to click on a webpage, which is loaded via pywb
https://github.com/webrecorder/pywb so that all content get archived for
later.
If I'm loading the page directly without pywb, I am able to click on a
botton on the page from python.
Trying to do the same with webrecorder, just errors out.
To Reproduce
1. Create a wayback-server with pywb under these instructions
https://pywb.readthedocs.io/en/latest/manual/usage.html#getting-started
2. Open a webpage from selenium with the archive
3. Try to click on the "Load earlier"-button via xpath
Expected behavior
On the Webpage, the button gets clicked and more comments are loaded.
(This is the case when the minds.com-url is directly loaded)
Actual behavior
Script just errors out and the button hasn't been clicked
<pre>
File "<stdin>", line 1, in <module>
File
"/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webelement.py",
line 80, in click
self._execute(Command.CLICK_ELEMENT)
File
"/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webelement.py",
line 633, in _execute
return self._parent.execute(command, params)
File
"/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py",
line 321, in execute
self.error_handler.check_response(response)
File
"/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py",
line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale
element reference: element is not attached to the page document
(Session info: chrome=70.0.3538.67)
(Driver info: chromedriver=2.41.578700
(2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 4.18.0-2-amd64
x86_64)
</pre>
Test script or set of commands reproducing this issue
pywb installation
<pre>
$ pip install pywb
$ wb-manager init archive
$ wayback --record --live -a --auto-interval 30
</pre>
### script connecting to pywb
<pre>
from selenium import webdriver
from time import sleep
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get("
http://localhost:8080/archive/record/https://www.minds.com/newsfeed/800266828058890240
")
(...)
print("click")
driver.find_element("xpath","//minds-comments/div/div").click()
print("sleep 3 sec")
sleep(3)
(...)
</pre>
Environment
OS: Debian 4.18.10-2
Browser: Chromium
Browser Driver version: chromedriver=2.41.578700
Language Bindings version: Python3 selenium==3.141.0