DrissionPage: Real Problems That Blocked Me (And How I Got Past Them)

Why I'm writing this

Used DrissionPage for a while after getting frustrated with Selenium. Tutorials made it look simple - "just click this button", "just scroll that page". Tried it on actual sites and hit walls immediately.

Looked at GitHub issues - turns out I wasn't alone. Lots of people hitting the same problems. Some solved, some still open.

This isn't a tutorial - it's me documenting the stuff that blocked me and how I got past it. Not perfect, not pretty, but it works for my use cases.

Sources: - DrissionPage GitHub Issues - Issue #329 - Cloudflare detection - Issue #654 - Multi-threading race conditions - Issue #659 - Scroll to bottom problem

Problem #1: Clicks don't register

What actually helped

from DrissionPage import ChromiumPage

page = ChromiumPage()
page.get('https://example.com')

# Problem: this doesn't actually click
button = page.ele('#submit-button')
button.click()

# What worked for me:
# 1. Wait longer before clicking
page.wait.load_start()

# 2. Make sure element is actually clickable
button = page.ele('#submit-button', timeout=10)
if button:
    # Click in different ways until one works
    try:
        button.click()
    except:
        # Try JS click
        button.run_js('this.click()')

    # Or click at center of element
    button.click(at=(0.5, 0.5))

Hidden elements

# Sometimes element is hidden by another element
# DrissionPage finds it but can't click it

# Solution: make element visible first
page.run_js('''
    const element = document.querySelector('#submit-button');
    if (element) {
        element.style.display = 'block';
        element.style.visibility = 'visible';
        element.style.opacity = '1';
    }
''')

# Then click
button.click()

# Check if anything is blocking clicks
def is_blocked(page, element):
    # Get element position
    rect = element.rect

    # Check if anything is covering it
    # (simplified - in reality this is more complex)
    covered = page.run_js(f'''
        const rect = arguments[0].getBoundingClientRect();
        const topEl = document.elementFromPoint(rect.x + rect.width/2, rect.y + rect.height/2);
        return topEl !== arguments[0];
    ''', element)

    return is_blocked(page, button)

Problem #2: Page scrolls to bottom automatically

Workarounds I tried

from DrissionPage import ChromiumPage
import time

page = ChromiumPage()
page.get('https://example.com')

# Problem: page auto-scrolls to bottom
# Solution 1: Wait for initial load, then scroll back up
page.wait(2)  # Let auto-scroll finish
scroll_to = 0  # scroll to top
page.run_js(f'window.scrollTo(0, {scroll_to});')

# Solution 2: Disable JS before page loads
# (doesn't work for all sites but worth trying)
page.get('https://example.com', timeout=30, load_mode='none')

# Solution 3: Run JS before page loads
page.run_js('''
    window.addEventListener('scroll', function(e) {
        e.stopPropagation();
        e.stopImmediatePropagation();
    }, true);
''')

# Solution 4: Set scroll position immediately after get
page.get('https://example.com', \
         lambda url: url.set.scroll_to_top())

What actually works reliably

# Intercept scroll events
page.get('https://example.com')

# Immediately set scroll position
page.run_js('window.scrollTo(0, 0);')

# Then wait for actual content to load
page.wait.doc_loaded()

# Force scroll position again
page.run_js('window.scrollTo(0, 0);')

# Sometimes need to do this multiple times
for _ in range(3):
    page.run_js('window.scrollTo(0, 0);')
    time.sleep(0.5)

Problem #3: Cloudflare bot detection

What someone in issue #329 tried

# User found pyautogui can do real mouse clicks
import pyautogui

# Get browser window position
# (need to manually measure or use tool)

# Click at specific coordinates
pyautogui.click(x, y)

# This worked for them but has drawbacks:
# - Requires knowing exact window position
# - Can't move mouse while script runs
# - Doesn't work headless

What worked for me

# Make browser look more human
from DrissionPage import ChromiumPage

page = ChromiumPage(
    arguments=[
        '--disable-blink-features=AutomationControlled',
        '--disable-dev-shm-usage',
        '--no-first-run',
        '--no-default-browser-check',
        '--disable-infobars',
    ],
    headless=False  # Headless is easier to detect
)

# Set realistic viewport
page.set.window.size(1920, 1080)
page.set.window.max()

# Add realistic user agent
page.set.user_agent(
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) '
    'AppleWebKit/537.36 (KHTML, like Gecko) '
    'Chrome/120.0.0.0 Safari/537.36'
)

Human-like delays

import random
import time

def human_click(element):
    """Click in a way that looks less automated"""
    # Move mouse to element first
    element.hover()

    # Random delay
    time.sleep(random.uniform(0.5, 1.5))

    # Click
    element.click()

    # Random delay after click
    time.sleep(random.uniform(1, 2))

# Also, don't click everything instantly
# Add random delays between actions
time.sleep(random.uniform(2, 5))

Problem #4: Multi-threading race conditions

The problem

from DrissionPage import ChromiumPage
import threading

def scrape(url):
    page = ChromiumPage()  # Creates new browser each time
    page.get(url)
    # ... do work
    page.quit()

# Problem with multiple threads
threads = []
for url in urls:
    t = threading.Thread(target=scrape, args=(url,))
    threads.append(t)
    t.start()

# Results: multiple browser windows
# Or: crashes due to race conditions

Solution that worked

from DrissionPage import ChromiumPage
from DrissionPage import ChromiumPage as Page
import threading

# Share single browser instance
# NOT recommended by DrissionPage docs but works

class BrowserPool:
    def __init__(self):
        self.page = ChromiumPage()
        self.lock = threading.Lock()

    def scrape(self, url):
        with self.lock:
            self.page.get(url)
            # ... do work
            # Don't quit, just navigate

# Use pool instead
pool = BrowserPool()
threads = []
for url in urls:
    t = threading.Thread(target=pool.scrape, args=(url,))
    threads.append(t)
    t.start()

for t in threads:
    t.join()

# When done
pool.page.quit()

Better approach: use multiprocessing

from DrissionPage import ChromiumPage
import multiprocessing

def scrape_worker(url):
    """Each process gets its own browser"""
    page = ChromiumPage()
    try:
        page.get(url)
        # ... do work
        return result
    finally:
        page.quit()

# Use processes instead of threads
with multiprocessing.Pool(processes=3) as pool:
    results = pool.map(scrape_worker, urls)

# Each process is isolated
# Slower but more stable
# Each has own browser instance

Problem #5: Element found but not interactable

Common scenario: element exists, DrissionPage finds it, but interaction fails.

Cause 1: Element in iframe

# Problem: can't find element in iframe
button = page.ele('#button')  # Returns None

# Solution: switch to iframe first
iframe = page.ele('iframe').src_frame

# Now search within iframe
button = iframe.ele('#button')

# Or select iframe by index
page.ele('iframe:nth-child(1)').src_frame

Cause 2: Shadow DOM

# Shadow DOM elements not accessible by default
# Solution: access through web component

page.get('https://example.com')

# Access shadow root
shadow_root = page.run_js('''
    return document.querySelector('my-component').shadowRoot;
''')

# Then search within shadow root
element = page.run_js('''
    const shadowRoot = document.querySelector('my-component').shadowRoot;
    return shadowRoot.querySelector('#button');
''')

# Or DrissionPage's shadow DOM support
from DrissionPage import WebPage

# Actually DrissionPage has some shadow DOM support
# but not perfect for all cases

Cause 3: Element not in viewport

# Element exists but outside viewport
element = page.ele('#button')

# Solution: scroll to element first
element.scroll.to_center()

# Or scroll into view
element.scroll.into_view()

# Then interact
element.click()

Problem #6: Dynamic content loading issues

Element exists but data hasn't loaded yet.

Explicit waits

from DrissionPage import ChromiumPage

page = ChromiumPage()
page.get('https://example.com')

# Don't immediately interact
# Wait for specific condition
def data_loaded(page):
    return page.ele('.data-loaded') is not None

# Wait up to 10 seconds
page.wait(10, data_loaded)

# Or wait for network to be idle
page.wait.network_idle()

# Then interact
data = page.ele('.content').text

Trigger lazy loading

# Some sites load content on scroll
# Need to scroll to trigger it

page.get('https://example.com')

# Scroll down slowly
for i in range(5):
    page.scroll.down(300)
    time.sleep(0.5)

# Now content is loaded
items = page.eles('.item')

# Scroll to load more
page.scroll.to_bottom()

# Wait for new content to appear
page.wait(2)  # wait for AJAX

# Get newly loaded items
new_items = page.eles('.item')

Problem #7: Cookies and session handling

Can't stay logged in, or cookies don't persist.

Cookie persistence

from DrissionPage import ChromiumPage

# Save cookies after login
page = ChromiumPage()
page.get('https://example.com/login')

# Do login stuff...

# Save cookies
cookies = page.cookies(as_dict=True)

# Save to file for later
import json
with open('cookies.json', 'w') as f:
    json.dump(cookies, f)

# Later, load cookies
with open('cookies.json', 'r') as f:
    cookies = json.load(f)

# Set cookies before navigating
for name, value in cookies.items():
    page.set.cookies(name, value)

# Now navigate
page.get('https://example.com/dashboard')  # Should be logged in

User data directory

# Use same user data directory to persist sessions
from DrissionPage import ChromiumPage

page = ChromiumPage(
    user_data_path='./browser_profile'
)

# First run: login, creates profile
page.get('https://example.com/login')
# do login...

# Next run: same path, session persists
page.quit()  # saves state

# Later...
page = ChromiumPage(
    user_data_path='./browser_profile'
)
page.get('https://example.com/dashboard')  # Still logged in!

How I debug when things go wrong

Debugging DrissionPage scripts can be frustrating. Here's what helps.

Visual debugging

# Don't use headless mode when debugging
page = ChromiumPage(headless=False)

# Watch what happens
# See the actual browser window

Screenshots on error

try:
    # ... do work ...
    element.click()
except Exception as e:
    # Take screenshot before failing
    screenshot_path = f'error_{int(time.time())}.png'
    page.get_screenshot(path=screenshot_path)
    raise

Check actual browser console

# See what JavaScript errors exist
console_errors = page.run_js('''
    return Array.from(window.console.errors).map(e => e.message);
''')

print("Console errors:", console_errors)

# Often reveals the real problem

Page source vs rendered HTML

# Sometimes page source differs from rendered HTML
# Check both

# What server sent
source = page.html

# What browser rendered (after JS)
rendered = page.run_js('document.body.innerHTML')

# Compare to find dynamically added content

Reality check

DrissionPage is great but it's not magic. Some sites just don't want to be automated. Cloudflare Enterprise, custom bot detection, obfuscated JavaScript - if a site really doesn't want you, they'll make it hard.

The issues I mentioned? They're from real use cases. Some I solved, some I worked around, some I just avoided those sites.

If you're hitting problems not covered here, check the GitHub issues. Someone probably hit it before. Search through closed issues too, not just open.

And if you find a solution that works, consider documenting it. Not for clout - but because I'll probably hit the same problem 6 months from now and forgot how I fixed it.

Good luck. Web automation is a cat-and-mouse game. DrissionPage helps but doesn't win every battle.