Why does pytest not execute the entire loop

[tejendrapsingh]

Hey! I am using the pytest together with seleniumbase for scraping a website. The function that I am passing to pytest has a long loop. pytest however stops before executing the entire loop. I have tried tweaking a few parameters but to no avail. I am posting a picture with a run that did not execute the entire loop. I am also posting the code below.

import pytest
import polars as pl
from itertools import chain
import os 
from pathlib import Path
import re
import traceback2 as traceback
from typing import Tuple
import sys

# Create test parameters that include both county and year
def generate_test_params():
    counties = ["oklahoma"]
    years = [2016]
    return [(county, year) for county in counties for year in years]

@pytest.mark.timeout(720000000)  # multiple hour timeout
@pytest.mark.parametrize("data,year_val", generate_test_params())
def test_multi_threaded(sb, data: str, year_val: int):
    """
    Test web scraping with extended timeout to allow for complete execution.
    Each worker handles a specific county-year combination to ensure completion.
    
    Args:
        sb: Selenium browser instance
        data: County name
        year_val: Year to process
    """
    try:
        # set the root directory
        os.chdir("C:/warrant_data_oklahoma")
        
        # Initialize counter at worker start
        no_data_counter = 0
        
        # check if the directory exists or not for saving the file
        directory = Path.cwd().joinpath(str(year_val), data)
        if directory.exists():
            print(f"Directory exists for {data}, {year_val}. Starting Scraping.")
        else:
            directory.mkdir(parents=True, exist_ok=True)
            print(f"Created directory for {data}, {year_val}. Starting Scraping.")
        
        # Set threshold based on county
        ndc_threshold = 500 if str(data)=='cotton' else 250
        
        # Process cases for this specific year-county combination
        for case_num in range(1, 20000):
            try:
                # Check threshold
                if no_data_counter > ndc_threshold:
                    print(f"Reached maximum number of cases for year {year_val} in county {data}")
                    # Instead of breaking, assert success to mark this worker's task as complete
                    assert True, f"Successfully completed {data} {year_val} after reaching threshold"
                    return
                
                # Create base filename pattern
                base_file_pattern = f"tr_{year_val}_{case_num}"
                file_types = ["overview", "parties", "attorneys", "events", "counts", "dockets", "defendant_info"]
                
                # Check if all required files exist
                files_exist = all(
                    Path.cwd().joinpath(str(year_val), data, f"{base_file_pattern}_{file_type}.csv").is_file()
                    for file_type in file_types
                )
                
                if files_exist:
                    print(f"All Files Exist. Skipping Scraping for File {case_num} in county {data}")
                    continue
                
                print(f"Starting Scraping for File {case_num} in county {data}, year {year_val}")
                url = f"https://www.oscn.net/dockets/GetCaseInformation.aspx?db={data}&number=TR-{year_val}-{case_num}"
                
                # Open URL with retry mechanism
                sb.uc_open_with_reconnect(url, reconnect_time=12)
                
                # Try to handle potential CAPTCHAs or submit buttons
                for xpath in [
                    '/html/body/div/div/div[1]/div/label/input',
                    "/html/body/div/form/input[1]"
                ]:
                    try:
                        sb.uc_click(xpath, by="xpath", reconnect_time=12)
                    except Exception:
                        print(f"Button click failed for xpath: {xpath}")

                # sometimes multiple records are found
                # Check for multiple records
                try:
                    sb.find_element(".multipleRecords")
                    multiple_records = True
                except:
                    multiple_records = False
                
                # Handle multiple records
                if multiple_records:
                    print(f"Multiple records found for case {case_num} in county {data}")
                    record_link = []
                    num_records = len(sb.find_elements(".multipleRecords > tbody > tr"))
                    for record_num in range(1, num_records + 1):
                        record_link.append(sb.get_attribute(
                            selector=f".multipleRecords > tbody > tr:nth-child({record_num}) > td:nth-child(1) > a",
                            attribute="href"
                        ))
                else:
                    record_link = [url]

                # loop over all records
                for case_link in record_link:
                    print(f"Starting Scraping for File {case_num} in county {data}")
                    sb.uc_open_with_reconnect(str(case_link), reconnect_time=12)
                    
                    # Handle verification steps
                    for xpath in [
                        '/html/body/div/div/div[1]/div/label/input',
                        "/html/body/div/form/input[1]"
                    ]:
                        try:
                            sb.uc_click(xpath, by="xpath", reconnect_time=12)
                        except:
                            print("Found no Submit Button. Starting Scraping.")

                    # Check case availability
                    try:
                        party_text = sb.get_text("#oscn-content > blockquote > font")
                    except:
                        party_text = ""
                        
                    if (re.search(f"Case Number/Document 'TR-{year_val}-{case_num}' is formatted incorrectly", party_text) or
                        re.search(f"TR-{year_val}-{case_num} not available at this time", party_text) or
                        re.search(r"is not found within "+data, party_text)):
                        print("No data available for this case. Skipping.")
                        no_data_counter += 1
                        continue
                    else:
                        # condition on the record being there
                        no_record_text = sb.get_text("#TABLE_1 > tbody > tr > td:nth-child(1)")
                        if re.search("No Record", no_record_text):
                            print("It appears that there is no data for this case. I am skipping scrapping.")
                        else:
                            # scrape overview table data
                            # save district court and county information    
                            county = sb.get_text("#oscn-content > div > h2.styletop > span")

                            # find number of rows in the overview table
                            num_rows = len(sb.find_elements("#TABLE_1 > tbody > tr"))
                            num_cols = len(sb.find_elements("#TABLE_1 > tbody > tr > td"))
                            # create an empty list to be filled later
                            overview_panel = []
                            # loop over each row and column
                            for row_num in range(1,num_rows+1):
                                # define empty list to be filled later
                                overview_panel_items = []
                                for col_num in range(1,num_cols+1):
                                    overview_panel_items.append(sb.get_text("#TABLE_1 > tbody > tr:nth-child("+str(row_num)+") > td:nth-child("+str(col_num)+")"))
                                # append to the list
                                overview_panel.append(overview_panel_items)
                            # work on the second element of the list to extract the citation number and other information
                            # I first split the string using new line operator; I then drop the empty strings within the returned list
                            overview_panel_right = [overview_panel[0][1].split("\n")][0]
                            # drop empty strings; these are rows in the second column without data
                            overview_panel_right = [x for x in overview_panel_right if x]
                            # extracting the column names and column values in a separate list
                            col_names = [x for x in [i.split(":")[0] for i in overview_panel_right] if x]
                            col_values = list(chain.from_iterable([[x for x in overview_panel_right if not(":") in x], [i.split(":")[1] for i in [x for x in overview_panel_right if ":" in x]]]))
                            # changing the first two elements of the column names list
                            col_names[0] = "citation_id"
                            col_names[1] = "offense_type"
                            # make a polars data frame
                            overview_panel = pl.DataFrame(
                                {
                                    "col_names" : col_names,
                                    "col_values" : col_values
                                }
                            )
                            # save the citation data to an object
                            citation_id = overview_panel.filter(pl.col("col_names") == "citation_id").select(pl.col("col_values"))
                            # add the citation ID column
                            overview_panel = overview_panel.with_columns(pl.lit(citation_id).alias("citation_id"))
                            # add the county variable
                            overview_panel = overview_panel.with_columns(pl.lit(county).alias("district_court_county"))
                            # save the data to the disk
                            overview_panel.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_overview.csv"))

                            # get parties information    
                            # note: sometimes between the overview table and parties there is a hyper link for whether the case is eligible for online payments or not
                            # note: this essentially changes the parties and attorneys xpath
                            # note: parties are always in a "p" element
                            # note: first "p" element is always the notice
                            # I need to condition on the occurence of this
                            # note: sometimes the second para has a child element which is a and sometimes not
                            try:
                                eligibility_check = sb.get_text('/html/body/div[1]/div[2]/div/p[2]/a')
                            except:
                                eligibility_check = sb.get_text('/html/body/div[1]/div[2]/div/p[2]')
                            if re.match("Case may be eligible for online payments", eligibility_check) or re.match("CASE MAY BE ELIGIBLE FOR ONLINE PAYMENTS", eligibility_check):
                                # this increases the counter of the parties paragraph
                                parties_para_counter = 3
                                parties_para_counter_span = 9
                            else:
                                parties_para_counter = 2
                                parties_para_counter_span = 8
                            print("Parties counter is "+str(parties_para_counter))
                            
                            # scrape information on the parties
                            parties = sb.get_text("/html/body/div[1]/div[2]/div/p["+str(parties_para_counter)+"]")
                            # split the string, trim whitespace and drop empty strings
                            parties = [x for x in [i.strip() for i in parties.split("\n")] if x]
                            # keep the characters after the last comma which are the parties
                            col_names = [i.split(", ")[-1] for i in parties]
                            # change the parties list to remove the party type at the end
                            parties = [i[:i.rfind(",")] for i in parties]
                            # generate a polars data frame
                            parties = pl.DataFrame(
                                {
                                    "party_type" : col_names,
                                    "party_name" : parties
                                }
                            )
                            # add the citation ID column
                            parties = parties.with_columns(pl.lit(citation_id).alias("citation_id"))
                            # add the county variable
                            parties = parties.with_columns(pl.lit(county).alias("district_court_county"))
                            # save data to the disk
                            parties.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_parties.csv"))

                            # scrape information on attorneys
                            # sometimes attorneys data is in a table format while at other times it is in "p" element
                            # I use the table ID to figure out if the attorneys data is in table format or not
                            # note: the first table is always the overview table; therefore if in table format attorneys data is the second table
                            # create a condition for attorney data being in a table format
                            try:
                                sb.get_text("#TABLE_2 > thead > tr > th:nth-child(1)")
                                attorneys_is_table = True
                            except:
                                attorneys_is_table = False
                            if attorneys_is_table is True:
                                if re.match("Attorney", sb.get_text("#TABLE_2 > thead > tr > th:nth-child(1)")):
                                    # create counts counter; just increase parties counter by one
                                    counts_para_counter = parties_para_counter + 1
                                    # if this evaluates to true, then the attorneys data is in a table format
                                    # get the table header as the column names
                                    # figure out the number of columns in the table
                                    num_cols = len(sb.find_elements("#TABLE_2 > thead > tr > th"))
                                    # create empty list to hold column names
                                    col_names = []
                                    # loop over each column name and append the empty list
                                    for col_num in range(1, num_cols+1):
                                        col_names.append(sb.get_text("#TABLE_2 > thead > tr > th:nth-child("+str(col_num)+")"))
                                    # get the table body
                                    # create empty list to be filled later
                                    attorneys = []
                                    # figure out the number of rows
                                    num_rows = len(sb.find_elements("#TABLE_2 > tbody > tr"))
                                    # loop over all the rows of the table body
                                    for row_num in range(1,num_rows+1):
                                        # create empty list to hold contents of each table row
                                        attorneys_item = []
                                        # loop over columns
                                        for col_num in range(1,num_cols+1):
                                            attorneys_item.append(sb.get_text("#TABLE_2 > tbody > tr:nth-child("+str(row_num)+") > td:nth-child("+str(col_num)+")"))
                                        # collect the items and append to the list
                                        attorneys.append(attorneys_item)
                                    # append attorneys data to have last row as the column names
                                    attorneys.append(col_names)    
                                    # create a polars data frame to hold the attorneys data
                                    attorneys = pl.DataFrame(attorneys,
                                                                orient="row")
                                    # store the count of rows in the data fram object; this is used to drop the last row of the attorneys data
                                    num_rows = attorneys.select(pl.len()).item()
                                    # rename the columns and drop last row
                                    attorneys = attorneys.rename(dict(zip(attorneys.columns, attorneys.tail(1).transpose().to_series()))).slice(0, num_rows-1)
                                    # add the citation ID column
                                    attorneys = attorneys.with_columns(pl.lit(citation_id).alias("citation_id"))
                                    # add the county variable
                                    attorneys = attorneys.with_columns(pl.lit(county).alias("district_court_county"))
                                    # save data to the disk
                                    attorneys.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_attorneys.csv"))
                                else:
                                    # create counts counter; increase parties counter by two
                                    counts_para_counter = parties_para_counter + 2
                                    # in this case the attorneys are not in table format
                                    attorneys = sb.get_text("/html/body/div[1]/div[2]/div/p["+str(parties_para_counter+1)+"]")
                                    # create a polars data frame
                                    attorneys = pl.DataFrame(
                                        {
                                            "Attorney" : attorneys
                                        }
                                    )
                                    # add the citation ID column
                                    attorneys = attorneys.with_columns(pl.lit(citation_id).alias("citation_id"))
                                    # add the county variable
                                    attorneys = attorneys.with_columns(pl.lit(county).alias("district_court_county"))
                                    # save data to the disk
                                    attorneys.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_attorneys.csv"))
                            elif attorneys_is_table is False:
                                # create counts counter; increase parties counter by two
                                counts_para_counter = parties_para_counter + 2
                                # in this case the attorneys are not in table format
                                attorneys = sb.get_text("/html/body/div[1]/div[2]/div/p["+str(parties_para_counter+1)+"]")
                                # create a polars data frame
                                attorneys = pl.DataFrame(
                                    {
                                        "Attorney" : attorneys
                                    }
                                )
                                # add the citation ID column
                                attorneys = attorneys.with_columns(pl.lit(citation_id).alias("citation_id"))
                                # add the county variable
                                attorneys = attorneys.with_columns(pl.lit(county).alias("district_court_county"))
                                # save data to the disk
                                attorneys.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_attorneys.csv"))
                            
                            # scrape events information
                            # create a condition for events table being there
                            try:
                                sb.find_element(".events_table")
                                events_table_exists = True
                            except:
                                events_table_exists = False
                            # condition on the table not being there and figuring out if the events has a "p" class
                            if events_table_exists is False:
                                try:
                                    sb.find_element(".section.events + p")
                                    events_is_para = True
                                    counts_para_counter = counts_para_counter + 1
                                except:
                                    events_is_para = False
                                    counts_para_counter = counts_para_counter
                            if events_table_exists is True:
                                # number of columns in the table
                                num_cols = len(sb.find_elements(".events_table > thead > tr > th"))
                                # create empty list to hold column names
                                col_names = []
                                # loop over each column name and append the empty list
                                for col_num in range(1, num_cols+1):
                                    col_names.append(sb.get_text(".events_table > thead > tr > th:nth-child("+str(col_num)+")"))
                                # get the table body
                                # create empty list to be filled later
                                events = []
                                # figure out the number of rows
                                num_rows = len(sb.find_elements(".events_table > tbody > tr"))
                                # loop over all the rows of the table body
                                for row_num in range(1,num_rows+1):
                                    # create empty list to hold contents of each table row
                                    events_item = []
                                    # loop over columns
                                    for col_num in range(1,num_cols+1):
                                        events_item.append(sb.get_text(".events_table > tbody > tr:nth-child("+str(row_num)+") > td:nth-child("+str(col_num)+")"))
                                    # collect the items and append to the list
                                    events.append(events_item)
                                # append events data to have last row as the column names
                                events.append(col_names)    
                                # create a polars data frame to hold the events data
                                events = pl.DataFrame(events, orient="row")
                                # store the count of rows in the data fram object; this is used to drop the last row of the data
                                num_rows = events.select(pl.len()).item()
                                # rename the columns and drop last row
                                events = events.rename(dict(zip(events.columns, events.tail(1).transpose().to_series()))).slice(0, num_rows-1)
                                # add the citation ID column
                                events = events.with_columns(pl.lit(citation_id).alias("citation_id"))
                                # add the county variable
                                events = events.with_columns(pl.lit(county).alias("district_court_county"))
                                # save data to the disk
                                events.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_events.csv"))
                            elif events_is_para is True:
                                print("Events are in a paragraph format for case "+str(case_num)+" in county "+str(data))
                                events = pl.DataFrame([sb.get_text(".section.events + p")])
                                events = events.rename(dict(zip(events.columns, ["count_description"])))
                                # add the citation ID column
                                events = events.with_columns(pl.lit(citation_id).alias("citation_id"))
                                # add the county variable
                                events = events.with_columns(pl.lit(county).alias("district_court_county"))
                                # save data to the disk
                                events.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_events.csv"))

                            # scrape counts information
                            # note: counts data will have two tables for each count
                            # note: sometimes the counts is not in table format
                            # note: I condition on the counts being in table
                            try:
                                sb.find_element(".CountsContainer > table")
                                counts_is_table = True
                            except:
                                counts_is_table = False
                            if counts_is_table is False:
                                try:
                                    sb.find_element("//h2[@class='section counts']/following-sibling::p", by = "xpath")
                                    counts_is_para = True
                                except:
                                    counts_is_para = False
                                try:
                                    sb.get_text("/html/body/div[1]/div[2]/div/p["+str(counts_para_counter)+"]")
                                    counts_is_para = True
                                except:
                                    counts_is_para = False
                            # condition on counts being in a table
                            if counts_is_table is True:
                                # print the message to the console
                                print("It appears that the counts are in a table format for case "+str(case_num)+" in county "+str(data))
                                # extract the number of counts
                                num_counts = len(sb.find_elements(".CountsContainer > table"))
                                # note: the first table for each count is the description of the count; the second table has disposition information
                                # empty list to hold each count
                                counts = []
                                # extract tables for each count
                                for count_num in range(1, num_counts+1):
                                    # extract count table
                                    # note: the table does not have header
                                    # figure out number of rows in the table body
                                    num_rows = len(sb.find_elements(".CountsContainer > table:nth-child("+str(1+((count_num-1)*2))+") > tbody > tr"))
                                    # loop over each row of the count table's body
                                    for row_num in range(1, num_rows+1):
                                        # create empty list to be filled later
                                        count_items = []
                                        # figure out the number of columns
                                        num_cols = len(sb.find_elements(".CountsContainer > table:nth-child("+str(1+((count_num-1)*2))+") > tbody > tr:nth-child("+str(row_num)+") > td"))
                                        # loop over each column
                                        for col_num in range(1, num_cols+1):
                                            # extract the text
                                            count_items.append(sb.get_text(".CountsContainer > table:nth-child("+str(1+((count_num-1)*2))+") > tbody > tr:nth-child("+str(row_num)+") > td:nth-child("+str(col_num)+")"))
                                        # convert the count description to polars object
                                        count_items = pl.DataFrame([count_items], orient="row")
                                        # rename the columns 
                                        count_items = count_items.rename(dict(zip(count_items.columns, ["count_number", "count_description"])))
                                    
                                    # extract party name and disposition information
                                    # extract the table header
                                    # figure out number of rows in the table header
                                    num_rows = len(sb.find_elements(".CountsContainer > table:nth-child("+str(2+((count_num-1)*2))+") > thead > tr"))
                                    # loop over each row of the table header
                                    for row_num in range(1, num_rows+1):
                                        # create an empty list that will be filled by the columns for each row
                                        col_items = []
                                        # figure out number of columns
                                        num_cols = len(sb.find_elements(".CountsContainer > table:nth-child("+str(2+((count_num-1)*2))+") > thead > tr:nth-child("+str(row_num)+") > th"))
                                        # loop over each column
                                        for col_num in range(1, num_cols+1):
                                            # extract the text
                                            col_items.append(sb.get_text(".CountsContainer > table:nth-child("+str(2+((count_num-1)*2))+") > thead > tr:nth-child("+str(row_num)+") > th:nth-child("+str(col_num)+")"))
                                        # assign to column names
                                        col_names = col_items    
                                    # extract table body
                                    # figure out number of rows in the table body
                                    num_rows = len(sb.find_elements(".CountsContainer > table:nth-child("+str(2+((count_num-1)*2))+") > tbody > tr"))
                                    # loop over each row of the table body
                                    for row_num in range(1, num_rows+1):
                                        # create an empty list that will be filled by the columns for each row
                                        col_items = []
                                        # figure out number of columns
                                        num_cols = len(sb.find_elements(".CountsContainer > table:nth-child("+str(2+((count_num-1)*2))+") > tbody > tr:nth-child("+str(row_num)+") > td"))
                                        # loop over each column
                                        for col_num in range(1, num_cols+1):
                                            # extract the text
                                            col_items.append(sb.get_text(".CountsContainer > table:nth-child("+str(2+((count_num-1)*2))+") > tbody > tr:nth-child("+str(row_num)+") > td:nth-child("+str(col_num)+")"))
                                        # convert to a polars object
                                        col_items = pl.DataFrame([col_items], orient="row")
                                        # rename the columns
                                        col_items = col_items.rename(dict(zip(col_items.columns, col_names)))
                                        # assign to a different name
                                        party_disposition = col_items
                                        # drop empty column
                                        party_disposition = party_disposition.drop("")

                                    # combine the two tables together
                                    count = pl.concat([count_items, party_disposition], how="horizontal")

                                    # append the count list
                                    counts.append(count)

                                # vertically stack the dataframe
                                counts = pl.concat(counts, how="vertical")

                                # add the citation ID column
                                counts = counts.with_columns(pl.lit(citation_id).alias("citation_id"))
                                # add the county variable
                                counts = counts.with_columns(pl.lit(county).alias("district_court_county"))
                                # save data to the disk
                                counts.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_counts.csv"))
                            elif counts_is_para is True:
                                print("Counts are not in table format for case "+str(case_num)+" in county "+str(data))
                                counts = pl.DataFrame([sb.get_text("/html/body/div[1]/div[2]/div/p["+str(counts_para_counter)+"]")])

                                # add the citation ID column
                                counts = counts.with_columns(pl.lit(citation_id).alias("citation_id"))
                                # add the county variable
                                counts = counts.with_columns(pl.lit(county).alias("district_court_county"))
                                # rename the first column
                                counts = counts.rename({"column_0" : "count_description"})
                                # save data to the disk
                                counts.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_counts.csv"))
                            else:
                                # print the message
                                print("It appears counts are not for case "+str(case_num)+" in county "+str(data))
                                # create a polars data object with just the citation identifier
                                counts = citation_id.clone()
                                # add the county variable
                                counts = counts.with_columns(pl.lit(county).alias("district_court_county"))
                                # save data to the disk
                                counts.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_counts.csv"))

                            # extract citation information
                            # note: sometimes the dockets data is not present
                            # note: I condition on the dockets being present
                            try:
                                sb.find_element("#oscn-content > div > blockquote")
                                citation_is_present = True
                            except:
                                citation_is_present = False
                            if citation_is_present is True:
                                citation = sb.get_text("#oscn-content > div > blockquote")
                                # create column names object
                                col_names = [y.strip() for y in [x.split(":")[0] for x in  [i.split("\n") for i in [citation]][0]]]
                                col_values = [y.strip() for y in [x.split(":")[1] for x in [i.split("\n") for i in [citation]][0]]]
                                # convert to polars dataframe
                                citation = pl.DataFrame([col_values], orient="row")
                                # rename columns
                                citation = citation.rename(dict(zip(citation.columns, col_names)))
                                # add the citation ID column
                                citation = citation.with_columns(pl.lit(citation_id).alias("citation_id"))
                                # add the county variable
                                citation = citation.with_columns(pl.lit(county).alias("district_court_county"))
                                # save data to the disk
                                citation.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_citation.csv"))
                            else:
                                print("It appears that there is no citation data for case "+str(case_num)+" in county "+str(data))
                                # create a polars data object with just the citation identifier
                                citation = citation_id.clone()
                                # add the county variable
                                citation = citation.with_columns(pl.lit(county).alias("district_court_county"))
                                # save data to the disk
                                citation.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_citation.csv"))


                            # extract dockets information
                            # note: dockets data is in table format
                            # note: dockets class changes for some counties
                            # condition on the dockets class matching
                            try:
                                sb.find_element(".docketlist.ocis")
                                docket_class_id = ".docketlist.ocis"
                                dockets_present = True
                            except:
                                dockets_present = False
                            if dockets_present is False:
                                try:
                                    sb.find_element(".docketlist.kp") 
                                    docket_class_id = ".docketlist.kp"
                                    dockets_present = True
                                except:
                                    dockets_present = False
                            if dockets_present is True:
                                # extract the table header
                                # figure out number of rows in the table header
                                num_rows = len(sb.find_elements(str(docket_class_id)+" > thead > tr"))
                                # loop over each row of the table header
                                for row_num in range(1, num_rows+1):
                                    # create an empty list that will be filled by the columns for each row
                                    col_items = []
                                    # figure out number of columns
                                    num_cols = len(sb.find_elements(str(docket_class_id)+" > thead > tr:nth-child("+str(row_num)+") > th"))
                                    # loop over each column
                                    for col_num in range(1, num_cols+1):
                                        # extract the text
                                        col_items.append(sb.get_text(str(docket_class_id)+" > thead > tr:nth-child("+str(row_num)+") > th:nth-child("+str(col_num)+")"))
                                    # assign to column names
                                    col_names = col_items

                                # generate polars object that will be expanded later
                                dockets = pl.DataFrame([col_names], orient="row")
                                dockets = dockets.rename(dict(zip(dockets.columns, col_names))).slice(1)
                                
                                # extract table body
                                # figure out number of rows in the table body
                                num_rows = len(sb.find_elements(str(docket_class_id)+" > tbody > tr"))
                                # loop over each row of the table body
                                for row_num in range(1, num_rows+1):
                                    # create an empty list that will be filled by the columns for each row
                                    col_items = []
                                    # figure out number of columns
                                    num_cols = len(sb.find_elements(str(docket_class_id)+" > tbody > tr:nth-child("+str(row_num)+") > td"))
                                    # loop over each column
                                    for col_num in range(1, num_cols+1):
                                        # extract the text
                                        col_items.append(sb.get_text(str(docket_class_id)+" > tbody > tr:nth-child("+str(row_num)+") > td:nth-child("+str(col_num)+")"))
                                    # convert to a polars object
                                    col_items = pl.DataFrame([col_items], orient="row")
                                    # rename the columns
                                    col_items = col_items.rename(dict(zip(col_items.columns, col_names)))
                                    # extend the dockets object created above
                                    dockets = pl.concat([dockets, col_items])

                                # add the citation ID column
                                dockets = dockets.with_columns(pl.lit(citation_id).alias("citation_id"))
                                # add the county variable
                                dockets = dockets.with_columns(pl.lit(county).alias("district_court_county"))
                                # save data to the disk
                                dockets.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_dockets.csv"))
                            else:
                                # create a polars data object with just the citation identifier
                                dockets = citation_id.clone()
                                # add the county variable
                                dockets = dockets.with_columns(pl.lit(county).alias("district_court_county"))
                                # save data to the disk
                                dockets.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_dockets.csv"))


                            # extract personal information
                            # I need to click the defendant link
                            # figure out where the defendant link is
                            parties = sb.get_text("/html/body/div[1]/div[2]/div/p["+str(parties_para_counter)+"]")
                            # condition on parties being there
                            if re.match("   None", parties):
                                print("No parties exist for case "+str(case_num)+" in county "+str(data))
                            else:
                                # we can extract the identifiers of the parties involved
                                # create parties index
                                parties_inv_index = [i for i, item in enumerate([i.split("\n") for i in [parties]][0]) if item] # this is enclosed within square brackets
                                parties_inv_index = [i+1 for i in parties_inv_index]
                                # create a flag for whether the parties data block has span tag or not
                                try:
                                    parties_span_tag_elements_len = len(sb.find_elements("#oscn-content > div > p:nth-child("+str(parties_para_counter_span)+") > span"))
                                except:
                                    parties_span_tag_elements_len=0
                                # sometimes there are no parties; condition on that
                                if len(parties_inv_index)>0:
                                    # empty list objects that will be filled later
                                    parties_id = []
                                    parties_desc = []
                                    # loop over all involved parties
                                    for parties_inv_num in parties_inv_index:
                                        # sometimes there is an extra empty line in the parties block; ignore the error raised while scraping this empty line
                                        # condition on whether there is span in the parties data block or not
                                        if parties_span_tag_elements_len>0:
                                            try:
                                                parties_id.append(sb.get_attribute("/html/body/div[1]/div[2]/div/p["+str(parties_para_counter)+"]/span["+str(parties_inv_num)+"]/a",by = "xpath",attribute = "href"))
                                                parties_desc.append(sb.get_text("/html/body/div[1]/div[2]/div/p["+str(parties_para_counter)+"]/span["+str(parties_inv_num)+"]/a"))
                                            except:
                                                print('Ran into an error while scraping parties information for party number '+str(parties_inv_num)+"for case "+str(case_num)+" in county "+str(data)+' in year '+str(year_val)+'. Could not locate the elements. Moving on.')
                                        else:
                                            try:
                                                parties_id.append(sb.get_attribute("/html/body/div[1]/div[2]/div/p["+str(parties_para_counter)+"]/a["+str(parties_inv_num)+"]",by = "xpath",attribute = "href"))
                                                parties_desc.append(sb.get_text("/html/body/div[1]/div[2]/div/p["+str(parties_para_counter)+"]/a["+str(parties_inv_num)+"]"))
                                            except:
                                                print('Ran into an error while scraping parties information for party number '+str(parties_inv_num)+"for case "+str(case_num)+" in county "+str(data)+' in year '+str(year_val)+'. Could not locate the elements. Moving on.')
                                    # convert to a polars object
                                    parties_inv_info=pl.DataFrame([parties_id,parties_desc]).rename(dict(zip(['column_0','column_1'],['parties_id','parties_desc'])))
                                    # add the citation ID column
                                    parties_inv_info = parties_inv_info.with_columns(pl.lit(citation_id).alias("citation_id"))
                                    # add the county variable
                                    parties_inv_info = parties_inv_info.with_columns(pl.lit(county).alias("district_court_county"))
                                    # save data to the disk
                                    parties_inv_info.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_parties_inv_info.csv"))
                                else:
                                    # create an object that is just the citation ID
                                    parties_inv_info = pl.DataFrame().with_columns(pl.lit(citation_id).alias("citation_id")).with_columns(pl.lit(county).alias("district_court_county"))
                                    # save data to the disk
                                    parties_inv_info.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_parties_inv_info.csv"))
                                    # print message to the console
                                    print('I could not locate any parties for case number '+str(case_num)+' in county '+data+' for year '+str(year_val)+'. Moving on to the next case. Hang tight.')

                                # extraction of defendant information            
                                defendant_index = [i for i, item in enumerate([i.split("\n") for i in [parties]][0]) if re.search("Defendant", item)] # this is enclosed within square brackets
                                defendant_index = [i+1 for i in defendant_index]
                                # sometimes there is no defendant information; condition on that
                                if len(defendant_index)>0:
                                    # empty list objects that will be filled later
                                    party_alias_all = []
                                    personal_info_all = []
                                    addresses_info_all = []
                                    # loop over all the defendants
                                    for defendant_num in defendant_index:
                                        # navigate to the URL
                                        sb.uc_open_with_reconnect(str(case_link),
                                                                        reconnect_time=12)
                                        try:
                                            sb.uc_click('/html/body/div/div/div[1]/div/label/input', by = "xpath", reconnect_time=12) # this clicks the submit button
                                        except:
                                            # do nothing
                                            print("Found no Submit Button. Starting Scraping.")     
                                        # check for the submit button; click it if found
                                        try:
                                            #sb.uc_switch_to_frame("iframe")
                                            #sb.uc_click("span.mark")
                                            sb.uc_click("/html/body/div/form/input[1]", by = "xpath", reconnect_time=12) # this clicks the submit button
                                        except:
                                            # do nothing
                                            print("Found no Submit Button. Continuing Scraping.")       
                                        # extract the link to the page and navigate there
                                        # condition on the parties data block having span tag elements
                                        if parties_span_tag_elements_len>0:
                                            p_info_link = sb.get_attribute("/html/body/div[1]/div[2]/div/p["+str(parties_para_counter)+"]/span["+str(defendant_num)+"]/a",
                                                                                by = "xpath",
                                                                                attribute = "href")
                                        else:
                                            p_info_link = sb.get_attribute("/html/body/div[1]/div[2]/div/p["+str(parties_para_counter)+"]/a["+str(defendant_num)+"]",
                                                                                by = "xpath",
                                                                                attribute = "href")
                                        print("I could find the link to the defendant page. Moving on to navigate to that page.")
                                        sb.uc_open_with_reconnect(str(p_info_link),
                                                                            reconnect_time=12)
                                        # try to click to verify human
                                        try:
                                            sb.uc_click('/html/body/div/div/div[1]/div/label/input', by = "xpath", reconnect_time=12) # this clicks the submit button
                                        except:
                                            # do nothing
                                            print("Found no Submit Button. Starting Scraping.")         
                                        # if the bot detection is activated, bypass it
                                        try:
                                            #sb.uc_switch_to_frame("iframe")
                                            #sb.uc_click("span.mark")
                                            sb.uc_click("/html/body/div/form/input[1]", by = "xpath", reconnect_time=12) # this clicks the submit button
                                        except:
                                            # do nothing
                                            print("Found no Submit Button. Continuing Scraping.")       

                                        # top table is the party name and aliases
                                        # figure out the number of rows in the header of the table
                                        num_rows = len(sb.find_elements(".partytable.partymain > thead > tr"))
                                        # loop over all rows
                                        for row_num in range(1, num_rows+1):
                                            # empty list that will be below
                                            col_items = []
                                            # figure out number of columns
                                            num_cols = len(sb.find_elements(".partytable.partymain > thead > tr:nth-child("+str(row_num)+") > th"))
                                            # loop over all columns to extract text and append the list
                                            for col_num in range(1, num_cols+1):
                                                col_items.append(sb.get_text(".partytable.partymain > thead > tr:nth-child("+str(row_num)+") > th:nth-child("+str(col_num)+")"))
                                            # assign to column names
                                            col_names = col_items
                                        # generate polars object that will be expanded later
                                        party_alias = pl.DataFrame([col_names], orient="row")
                                        party_alias = party_alias.rename(dict(zip(party_alias.columns, col_names))).slice(1)
                                        # figure out number of rows in the table body
                                        num_rows = len(sb.find_elements(".partytable.partymain > tbody > tr"))
                                        # loop over each row of the table body
                                        for row_num in range(1, num_rows+1):
                                            # create an empty list that will be filled by the columns for each row
                                            col_items = []
                                            # figure out number of columns
                                            num_cols = len(sb.find_elements(".partytable.partymain > tbody > tr:nth-child("+str(row_num)+") > td"))
                                            # loop over each column
                                            for col_num in range(1, num_cols+1):
                                                # extract the text
                                                col_items.append(sb.get_text(".partytable.partymain > tbody > tr:nth-child("+str(row_num)+") > td:nth-child("+str(col_num)+")"))
                                            # convert to a polars object
                                            col_items = pl.DataFrame([col_items], orient="row")
                                            # rename the columns
                                            col_items = col_items.rename(dict(zip(col_items.columns, col_names)))
                                            # extend the party_alias object created above
                                            party_alias = pl.concat([party_alias, col_items])
                                        # reshape to wide
                                        # generate the index variable
                                        party_alias = party_alias.with_columns(pl.Series("party_index", list(range(1, len(party_alias)+1))))
                                        # add citation ID column
                                        party_alias = party_alias.with_columns(pl.lit(citation_id).alias("citation_id"))
                                        # do reshaping
                                        party_alias = party_alias.pivot(index = "citation_id",
                                                                        values = [x for x in party_alias.columns if not(re.match("party_index", x)) and not(re.match("citation_id", x))],
                                                                        on = "party_index",
                                                                        separator="_party_index_")
                                        # drop citation ID column
                                        party_alias = party_alias.drop("citation_id")
                                        # append the list
                                        party_alias_all.append(party_alias)

                                        # middle table is personal profile
                                        # figure out the number of rows in the header of the table
                                        num_rows = len(sb.find_elements(".partytable.personal > thead > tr"))
                                        # loop over all rows
                                        for row_num in range(1, num_rows+1):
                                            # empty list that will be below
                                            col_items = []
                                            # figure out number of columns
                                            num_cols = len(sb.find_elements(".partytable.personal > thead > tr:nth-child("+str(row_num)+") > th"))
                                            # loop over all columns to extract text and append the list
                                            for col_num in range(1, num_cols+1):
                                                col_items.append(sb.get_text(".partytable.personal > thead > tr:nth-child("+str(row_num)+") > th:nth-child("+str(col_num)+")"))
                                            # assign to column names
                                            col_names = col_items
                                        # generate polars object that will be expanded later
                                        personal_info = pl.DataFrame([col_names], orient="row")
                                        personal_info = personal_info.rename(dict(zip(personal_info.columns, col_names))).slice(1)
                                        # figure out number of rows in the table body
                                        num_rows = len(sb.find_elements(".partytable.personal > tbody > tr"))
                                        # loop over each row of the table body
                                        for row_num in range(1, num_rows+1):
                                            # create an empty list that will be filled by the columns for each row
                                            col_items = []
                                            # figure out number of columns
                                            num_cols = len(sb.find_elements(".partytable.personal > tbody > tr:nth-child("+str(row_num)+") > td"))
                                            # sometimes there is no record found
                                            # I condition on this
                                            if num_cols == 1 and re.search("No Records Found", sb.get_text(".partytable.personal > tbody > tr:nth-child(1) > td:nth-child(1)")):
                                                print("It appears no record is found")
                                                col_items = ["No Record Found"]*len(col_names)
                                            else:
                                                # loop over each column
                                                for col_num in range(1, num_cols+1):
                                                    # extract the text
                                                    col_items.append(sb.get_text(".partytable.personal > tbody > tr:nth-child("+str(row_num)+") > td:nth-child("+str(col_num)+")"))
                                            # convert to a polars object
                                            col_items = pl.DataFrame([col_items], orient="row")
                                            # rename the columns
                                            col_items = col_items.rename(dict(zip(col_items.columns, col_names)))
                                            # extend the personal_info object created above
                                            personal_info = pl.concat([personal_info, col_items])
                                        # reshape to wide
                                        # generate the index variable
                                        personal_info = personal_info.with_columns(pl.Series("profile_index", list(range(1, len(personal_info)+1))))
                                        # add citation ID column
                                        personal_info = personal_info.with_columns(pl.lit(citation_id).alias("citation_id"))
                                        # do reshaping
                                        personal_info = personal_info.pivot(index = "citation_id",
                                                                            values = [x for x in personal_info.columns if not(re.match("profile_index", x)) and not(re.match("citation_id", x))],
                                                                            on = "profile_index",
                                                                            separator="_profile_index_")
                                        # drop citation ID column
                                        personal_info = personal_info.drop("citation_id")
                                        # append to the list object
                                        personal_info_all.append(personal_info)

                                        # third and final table is the addresses table
                                        # figure out the number of rows in the header of the table
                                        num_rows = len(sb.find_elements(".partytable.addresses > thead > tr"))
                                        # loop over all rows
                                        for row_num in range(1, num_rows+1):
                                            # empty list that will be below
                                            col_items = []
                                            # figure out number of columns
                                            num_cols = len(sb.find_elements(".partytable.addresses > thead > tr:nth-child("+str(row_num)+") > th"))
                                            # loop over all columns to extract text and append the list
                                            for col_num in range(1, num_cols+1):
                                                col_items.append(sb.get_text(".partytable.addresses > thead > tr:nth-child("+str(row_num)+") > th:nth-child("+str(col_num)+")"))
                                            # assign to column names
                                            col_names = col_items
                                        # generate polars object that will be expanded later
                                        addresses_info = pl.DataFrame([col_names], orient="row")
                                        addresses_info = addresses_info.rename(dict(zip(addresses_info.columns, col_names))).slice(1)
                                        # figure out number of rows in the table body
                                        num_rows = len(sb.find_elements(".partytable.addresses > tbody > tr"))
                                        # loop over each row of the table body
                                        for row_num in range(1, num_rows+1):
                                            # create an empty list that will be filled by the columns for each row
                                            col_items = []
                                            # figure out number of columns
                                            num_cols = len(sb.find_elements(".partytable.addresses > tbody > tr:nth-child("+str(row_num)+") > td"))
                                            # sometimes there is no record found
                                            # I condition on this
                                            if num_cols == 1 and re.search("No Records Found", sb.get_text(".partytable.addresses > tbody > tr:nth-child(1) > td:nth-child(1)")):
                                                print("It appears no record is found")
                                                col_items = ["No Record Found", "No Record Found", "No Record Found", "No Record Found"]
                                            else:
                                                # loop over each column
                                                for col_num in range(1, num_cols+1):
                                                    # extract the text
                                                    col_items.append(sb.get_text(".partytable.addresses > tbody > tr:nth-child("+str(row_num)+") > td:nth-child("+str(col_num)+")"))
                                            # convert to a polars object
                                            col_items = pl.DataFrame([col_items], orient="row")
                                            # rename the columns
                                            col_items = col_items.rename(dict(zip(col_items.columns, col_names)))
                                            # extend the addresses_info object created above
                                            addresses_info = pl.concat([addresses_info, col_items])
                                        # reshape to wide
                                        # generate the index variable
                                        addresses_info = addresses_info.with_columns(pl.Series("address_index", list(range(1, len(addresses_info)+1))))
                                        # add citation ID column
                                        addresses_info = addresses_info.with_columns(pl.lit(citation_id).alias("citation_id"))
                                        # do reshaping
                                        addresses_info = addresses_info.pivot(index = "citation_id",
                                                                            values = [x for x in addresses_info.columns if not(re.match("address_index", x)) and not(re.match("citation_id", x))],
                                                                            on = "address_index",
                                                                            separator="_address_index_")
                                        # drop citation ID column
                                        addresses_info = addresses_info.drop("citation_id")
                                        # append to the list 
                                        addresses_info_all.append(addresses_info)

                                    # convert to a polars object
                                    party_alias = pl.concat(party_alias_all, how = "diagonal")
                                    personal_info = pl.concat(personal_info_all, how = "diagonal")
                                    addresses_info = pl.concat(addresses_info_all, how = "diagonal")

                                    # merge the three together
                                    defendant_info = pl.concat([party_alias, personal_info, addresses_info], how = "horizontal")
                                    # add the citation ID column
                                    defendant_info = defendant_info.with_columns(pl.lit(citation_id).alias("citation_id"))
                                    # add the county variable
                                    defendant_info = defendant_info.with_columns(pl.lit(county).alias("district_court_county"))
                                    # save data to the disk
                                    defendant_info.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_defendant_info.csv"))
                                else:
                                    # create an object that is just the citation ID
                                    defendant_info = pl.DataFrame().with_columns(pl.lit(citation_id).alias("citation_id")).with_columns(pl.lit(county).alias("district_court_county"))
                                    # save data to the disk
                                    defendant_info.write_csv(Path.cwd().joinpath(str(year_val), data, "tr_"+str(year_val)+"_"+str(case_num)+"_defendant_info.csv"))
                                    # print message to the console
                                    print('I could not locate any defendant for case number '+str(case_num)+' in county '+data+' for year '+str(year_val)+'. Moving on to the next case. Hang tight.')    
                
            except Exception as e:
                print(f'Scraping Failed for case {case_num} in {data} {year_val}: {str(e)}')
                print("*** print_exception:")
                no_data_counter += 1
                continue
        
        # If we complete the loop normally, assert success
        assert True, f"Successfully completed all cases for {data} {year_val}"
        
    except Exception as e:
        print("*** print_exception:")
        pytest.fail(f"Test failed for {data} {year_val} with error: {str(e)}")

# Add conftest.py configuration
def pytest_configure(config):
    config.addinivalue_line(
        "markers",
        "timeout: mark test to set a timeout value in seconds"
    )

[graingert]

pytest with xdist doesn't print anything out when using -s pytest-dev/pytest-xdist#354

Probably this issue is in where you catch all exceptions and suppress them https://github.com/tejendrapsingh/for-help/blob/main/using-pytest-for-parallelization.py#L900

Instead you should only catch the exceptions that you expect

If you want to catch multiple exceptions and continue a for loop you should use an exception group:

exceptions = []
for x in y:
    try:
        ...
    except Exception as e:
        e.add_note("failed processing {x}")
        exceptions.append(e)
if exceptions:
    raise ExceptionGroup("multiple errors processing", exceptions)

These bare except statements are also bad, and incompatible with a signal handler and BaseException timeout approach: https://github.com/tejendrapsingh/for-help/blob/main/using-pytest-for-parallelization.py#L92

[graingert]

Also this is not the correct way to add metadata to an exception:

except Exception as e:
        print("*** print_exception:")
        pytest.fail(f"Test failed for {data} {year_val} with error: {str(e)}")

You should instead do:

except Exception as e:
    e.add_note(f"Test failed for {data} {year_val}")
    raise

[tejendrapsingh]

@graingert Thanks for these suggestions. I will edit the code with what you suggested and see if it makes a difference. Thank you very much for taking the time to go through the code. I am not a native programmer. So, always trying to find the easiest way to make things work. Pardon my ``dirty'' code writing.

[The-Compiler]

I'd suggest you don't use pytest for this at all, as you don't actually seem to be testing anything. If you use Selenium directly from Python, and then use something like the concurrent.futures module to parallelize that, you'll have much more control over what's happening.

I also agree that your approach to exception handling likely hides bugs in your code - see e.g. The Most Diabolical Python Antipattern – Real Python.