Debug School

rakesh kumar
rakesh kumar

Posted on • Updated on

Selenium webscraping:common mistake

First

driver=webdriver.chrome()

solution
use capital c in chrome
driver=webdriver.Chrome()

Second

WebDriverException

Solution
Run again

import pandas as pd
from selenium import webdriver
import warnings
warnings.filterwarnings('ignore')
from selenium.webdriver.common.by import By
import time
Enter fullscreen mode Exit fullscreen mode 
driver=webdriver.Chrome()
Enter fullscreen mode Exit fullscreen mode

Another way

wait = WebDriverWait(driver, 10)
element = wait.until(EC.presence_of_element_located((By.ID, "element-id")))
Enter fullscreen mode Exit fullscreen mode 

driver=webdriver.Chrome()
driver.get('https://www.amazon.in/')
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
search = wait.until(EC.presence_of_element_located((By.XPATH, '//*[@id="twotabsearchtextbox"]')))
search.send_keys('Guitar')
Enter fullscreen mode Exit fullscreen mode

Third

ValueError: All arrays must be of the same length

Solution
run single colunm

import pandas as pd
df= pd.DataFrame({'company_name':company_name})
df
Enter fullscreen mode Exit fullscreen mode

Fourth

NameError: name 'driver' is not defined
solution

u have to run command from initially
import pandas as pd
from selenium import webdriver
import warnings
warnings.filterwarnings('ignore')
from selenium.webdriver.common.by import By
import time
Enter fullscreen mode Exit fullscreen mode 
driver=webdriver.Chrome()
Enter fullscreen mode Exit fullscreen mode 
driver.get('https://www.flipkart.com/')
Enter fullscreen mode Exit fullscreen mode

Five 

TypeError: 'WebElement' object is not iterable
Enter fullscreen mode Exit fullscreen mode

reason
The error you're encountering is because you're trying to iterate over a single WebElement object, which is not iterable.

all_brand=driver.find_element(By.CLASS_NAME,"_2WkVRV")
all_brand
Enter fullscreen mode Exit fullscreen mode

output

<selenium.webdriver.remote.webelement.WebElement (session="3873cbd5e3f8b42981652a5f9c988eec", element="A3A6384478CB7478A3A428DE530AFC33_element_178")>
Enter fullscreen mode Exit fullscreen mode

i use driver.find.element but actually i want to use driver.find_elements

solution
use elements not element

all_brand=driver.find_elements(By.CLASS_NAME,"_2WkVRV")
all_brand
Enter fullscreen mode Exit fullscreen mode

six:

AttributeError: 'WebElement' object has no attribute 'append'
my code

for cost in all_price:  

    price.append(cost.text)
price
Enter fullscreen mode Exit fullscreen mode

Error

AttributeError                            Traceback (most recent call last)
Cell In[32], line 4
      1 mypri=[]
      2 for cost in all_price:  
----> 4     price.append(cost.text)
      5 price

AttributeError: 'WebElement' object has no attribute 'append'
Enter fullscreen mode Exit fullscreen mode

Reason
It seems like you're encountering an AttributeError because you're trying to append a WebElement directly to a list. To extract text from WebElement objects and append them to a list, you should call the .text attribute on each WebElement and then append the resulting text to the list.

Solution

price=[]
for cost in all_price:  

    price.append(cost.text)
price
Enter fullscreen mode Exit fullscreen mode

output

['₹279',
 '₹279',
 '₹277',
 '₹822',
 '₹1,558',
 '₹622',
 '₹4,000',
 '₹1,285',
 '₹658',
 '₹1,679',
 '₹499',
 '₹1,485',
 '₹749',
 '₹279',
 '₹749',
 '₹1,766',
 '₹499',
 '₹1,264',
 '₹1,285',
 '₹587',
 '₹699',
 '₹695',
 '₹3,330',
 '₹549',
 '₹622',
 '₹646']
Enter fullscreen mode Exit fullscreen mode

Seven

AttributeError: 'str' object has no attribute 'text'
my code

 for description in tags_div:
        anchor_tags = description.find_elements(By.TAG_NAME, 'a')


        tag_text = ', '.join(tag.text for tag in anchor_tags)
        desc.append(tag_text.text)
Enter fullscreen mode Exit fullscreen mode

error

AttributeError                            Traceback (most recent call last)
Cell In[59], line 25
     21     anchor_tags = description.find_elements(By.TAG_NAME, 'a')
     24     tag_text = ', '.join(tag.text for tag in anchor_tags)
---> 25     desc.append(tag_text.text)
     26 # Try to find and click the element that loads more items (e.g., a "Load More" button)
     27 next_button = None

AttributeError: 'str' object has no attribute 'text'
Enter fullscreen mode Exit fullscreen mode

solution

    for description in tags_div:
        anchor_tags = description.find_elements(By.TAG_NAME, 'a')


        tag_text = ', '.join(tag.text for tag in anchor_tags)
        desc.append(tag_text)//tag_text is used here
Enter fullscreen mode Exit fullscreen mode

output

Image description

Eight 

AttributeError                            Traceback (most recent call last)
Cell In[17], line 1
----> 1 designation.send_keys('Data Analyst')

AttributeError: 'list' object has no attribute 'send_keys'
Enter fullscreen mode Exit fullscreen mode

solution

The error message you encountered, AttributeError: 'list' object has no attribute 'send_keys', indicates that you're trying to use the send_keys method on a list object (designation) instead of an element.

To resolve this issue, you should ensure that you're working with the correct WebElement object before calling send_keys. Here's a general outline of how to find an element and then send keys to it:

Use one of the find_element methods to locate the input element you want to interact with.

Once you have the WebElement object representing the input element, you can call the send_keys method on it.

Here's an example:

from selenium import webdriver

# Start a WebDriver instance (e.g., Firefox) and navigate to a webpage
driver = webdriver.Firefox()
driver.get("https://example.com")  # Replace with the URL of your webpage

try:
    # Find the input element by its name (you can use other locators like ID, XPath, etc.)
    designation_input = driver.find_element_by_name("designation")

    # Send keys to the input element
    designation_input.send_keys("Data Analyst")

    # Wait for a moment to observe the input (optional)
    driver.implicitly_wait(2)
except Exception as e:
    # Handle any exceptions that may occur during execution
    print("An error occurred:", str(e))
finally:
    # Close the browser window
    driver.quit()
Enter fullscreen mode Exit fullscreen mode

In this example:

We start a WebDriver instance (Firefox) and navigate to a webpage.

We locate the input element with the name "designation" using driver.find_element_by_name("designation").

We then use the send_keys method on the designation_input WebElement object to send the keys "Data Analyst" to the input field.

Make sure to replace "https://example.com" with the URL of the webpage you're working with and use an appropriate locator method to find the input element on your page.

Nine 

NameError
Enter fullscreen mode Exit fullscreen mode

Traceback (most recent NameError Traceback (most recent call last)
Cell In[15], line 1
----> 1 designationinput.send_keys('Guitar')

NameError: name 'designationinput' is not definedll last)
Cell In[15], line 1
----> 1 designationinput.send_keys('Guitar')

NameError: name 'designationinput' is not defined

my code is

search = wait.until(EC.presence_of_element_located((By.XPATH, '//*[@id="twotabsearchtextbox"]')))

designationinput.send_keys('Guitar')
Enter fullscreen mode Exit fullscreen mode

here variable name is search but key is sended by designationinput
(designationinput.send_keys('Guitar') so correct code is
solution

search.send_keys('Guitar')
Enter fullscreen mode Exit fullscreen mode

TypeError Traceback (most recent call last)
Cell In[66], line 7
2 for page_number in range(1, 6): # Change '6' to the number of pages you want to search
3 # Your code to scrape/search on the current page goes here
4
5 # Check for the "Next" button to go to the next page
6 next_button = driver.find_element(By.XPATH, '//*[@id="search"]/div[1]/div[1]/div/span[1]/div[1]/div[3]/div/div/div/div/div/div/div[2]/div[2]/h2')
----> 7 for i in next_button:
8 head_url.append(i.text)
9 head_url.text

TypeError: 'WebElement' object is not iterable

SEARCHING TECHNIQUE

FIRST
my code is ram = driver.find_element(By.XPATH, '//div[@class="_2418kt"]/ul/li[1]')
ram_text = ram.text
ram_text and output '8 GB RAM | 128 GB ROM' but i want only 8gb ram

SECOND
u have to get scrape all phone detail then first

driver=webdriver.Chrome()
driver.get('https://www.flipkart.com/oneplus-nord-ce-3-lite-5g-chromatic-gray-128-gb/p/itm2cd5a4e659035?pid=MOBGZJ42KHUZZKMN&lid=LSTMOBGZJ42KHUZZKMNI4CBJY&marketplace=FLIPKART&q=oneplus+nord&store=tyy%2F4io&srno=s_1_5&otracker=AS_Query_HistoryAutoSuggest_1_3_na_na_na&otracker1=AS_Query_HistoryAutoSuggest_1_3_na_na_na&fm=organic&iid=fb5d3568-3f97-4497-850e-7288038a51c2.MOBGZJ42KHUZZKMN.SEARCH&ppt=hp&ppn=homepage&ssid=l83adrtetc0000001696041804960&qH=e8b736bf4fa84421/')

then apply select element by id/class

==========================================

how to get xpath

step 1: first inspect then click right on 3 dots

Image description

step2: take first header xpath

/html/body/div[1]/div[4]/main/div[3]/div/div[2]/article[1]/h2/a/span
Enter fullscreen mode Exit fullscreen mode

step3 take second header xpath

/html/body/div[1]/div[4]/main/div[3]/div/div[2]/article[2]/h2/a/span
Enter fullscreen mode Exit fullscreen mode

step4: which is not common between two xpath

article[1] and article[2]  are not common
Enter fullscreen mode Exit fullscreen mode

step 5 take common things to get final xpath

article
Enter fullscreen mode Exit fullscreen mode 
/html/body/div[1]/div[4]/main/div[3]/div/div[2]/article/h2/a/span
Enter fullscreen mode Exit fullscreen mode

Top comments (0)